5095 Servers FPS Lag

Is there a way to fix the lag for 5095 clients? It's NOT my computer, the game runs fine and everything but when I'm in an area with more than 3 people, my FPS goes from 30-173 to 173-28, jumps like crazy, is there a fix to this?