Ensure there are no collisions, this can be checked with ifconfig or ethtool. Collisions are caused by other devices on the network sending traffic at the same time as the server you are diagnosing. Collisions are normal at a low rate, however higher rates of collisions are typicaly caused by faulty NIC cards or poorly terminated cables.

Port Ranges

If your machine requires a high volume of connections, ensure there are enough ports availible for this traffic. For example and apache box, can produce a number of connections to tomcat, aswell as connections to an nfs per any one web request. Each of these connections will require an availible port.

To check your current port range, use the command:
cat /proc/sys/net/ipv4/ip_local_port_range

Increase the RAM dedicated to Networking

The default in linux is to allow the kernel to allocate ram dynamically to networking. The default setup is fine so tcp_mem can be left alone, however you can configure the default memory allocation aswell as increasing the maximum allowed memory usage for networking. The default value should never exceed the maximum value. The linux kernel does set the defaults up based on the installed memory, however these calculations are not based on your network traffic.

to check the memory allocated to networking you can use;
cat /proc/sys/net/ipv4/tcp_mem

to check the recieving socket memory allocations;
cat /proc/sys/net/core/rmem_default
cat /proc/sys/net/core/rmem_max

to check the sending socket memory allocations;
cat /proc/sys/net/core/wmem_default
cat /proc/sys/net/core/wmem_max

these configurations can be edited in the sysctl configuration file. The allocations are specified in bytes. The following will work well for gigabit ethernet (16MB allocations);

net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.core.optmem_max = 40960
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

TCP Timestamps

TCP Timestamps are useful however from a security perspective it's recommended to switch this off of external facing servers.

cat /proc/sys/net/ipv4/tcp_sack

set this to 1 for internal servers, and 0 for external facing servers.

Disable TCP Slow Start

To check for TCP slow start;
cat /proc/sys/net/ipv4/tcp_slow_start_after_idle

To disable use the following sysctl config;
net.ipv4.tcp_slow_start_after_idle = 0

Add additional allocation space for TIME_WAIT sockets

add the following config to sysctl to enable more TIME_WAIT sockets to be open and for clients to reconnect using the same socket.

net.core.netdev_max_backlog = 50000
net.ipv4.tcp_max_syn_backlog = 30000
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 10

Speeding up UDP

if your nfs uses udp or you use udp intensiveley, increase the udp memory allocation;
net.ipv4.udp_rmem_min = 8192
net.ipv4.udp_wmem_min = 8192

Disable Source Routing and Redirects

This protects from malicios requests like icmp redirects;

net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.accept_source_route = 0

TCP Congestion Avoidance Algorithm

The default algorithm in linux to prevent tcp congestion used to be RENO, this is also used in a number of other unix based systems. Most newer distros using kernel version 2.6.19 are migrating over to CUBIC.

you can check the algorithm you are using with;
cat /proc/sys/net/ipv4/tcp_congestion_control

and once again configuring this in sysctl. In most cases CUBIC will work fine, however if you are expeciencing alot of congestion, it can be worth while benchmarking others.