Drop new TCP connection from single IP if it exceeds 10 simultaneous connections.
iptables -I INPUT 1 -p tcp --syn -m connlimit --connlimit-above 10 -j DROP