WOA!! you have to realize what that is!
Networking protocol has a memory overhead. At any time your network card (in memory) is buffering out going packets, reading in coming and transfering them to the appropriate applications, and it must keep a cache of sent packets. The sent packet cache is for when the recipient does not receive, or get a corrupted packet and they must re-request it.
All that remains in your computer's memory. If you want to reduce the usage, just kill your network traffic. Push people and users off the box (sarcasm)
Your culprits are net idiots that either over-load their network connections, or the unfortunate few who still have dial-up or other slow connections. Because these transmissions go slower, they reside in memory longer and need more buffering.
If you don't have it, try modules like limitIPconn to reduce the amount of connections net-idiots start. A respectible value for multiple connection is from 12, to even 24. I have logs showing users creating as many as 128 concurrent connections to get their stuff "faster"... Everything this [ab]user requests then remains in the server's memory until their connection can complete all the downloads.