Why do I only get 300-400 Mbit/s on my 1000 Mbit/s network?

"Jeff Atwood has some good analysis of why real-world throughput of a Gigabit network is closer to, in his calculations, 30MB/s. (big 'B', not little 'b')

...you definitely shouldn't expect the perfect scaling we achieved moving from 10baseT to 100baseT. Without any major tweaking, you'll get only a fraction of the tenfold bandwidth improvement you might expect"

"We've performed numerous test and it boiled down to the Operating System.

If we go from Ubuntu 11.04 to Ubuntu 11.04 (same with CentOS for Linux machines), the speed throughput is ~938-941 MBits/s constant. If we go from Windows 7 x64 SP1 Ultimate to Ubuntu 11.04, it varies from ~440-475 MBits/s. If we move from Windows 2008 R2SP1 x64 to Ubuntu 11.04, the speed measures ~435-635 MBits/s.

As you can see, Linux based machines fully utilize the Gigabit links compared to Windows. We are now investigating settings in Windows that would result in the same throughput as Linux on same hardware. Hard drive limitations are not a factor since this is in memory. For the record, the machines we used had 7200 RPM drives with SATA 3.0. Both the Ubuntu 11.04 (or Linux base) and Windows exceeded the theoretical speed of those SATA links."

"Transfer speeds are only as good as the weakest link in the chain. It might not be the network speed but the write/read speed of your Hard Drives.

You can download RamDisk software and create a RamDisk on both computers. Place a decent size file inside of it and do a transfer over the network between the RamDisks. This should show you the actual speed of your network that won't be bottlenecked by your Hard Drive speeds."

"Not a direct answer to your question but recently we got the following result with iperf between two Linux boxes (8 core, 2.66 GHz each):

------------------------------------------------------------
Client connecting to 192.168.74.15, TCP port 5001
TCP window size: 192 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.74.25 port 54016 connected with 192.168.74.15 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 1.10 GBytes 941 Mbits/sec

with MTU=1500 (i.e. not using jumbo frames). That should give you an idea of what performance is in principle possible.

The network cards (lspci output) are:

82571EB Gigabit Ethernet Controller

on both sides.

You might want to check the CPU load on the server and especially on the client side and see whether this is close to 100%. Even though I have to say that on our system the busiest core is 80% (83%) idle on the receiver (sender) side while the other 7 seem to be 100% idle (checked with mpstat).

This is running iperf without the -u option, i.e. this uses TCP."