Rigs of Rods

I was playing MP last night in .38.22 with a friend. As soon as a third person joined, with a vehicle we both could see, we both experienced severe framerate stutter/lag (from 60fps vsync locked to 20-30 and stuttering). It had nothing to do with physics or graphical calculation, because it was doing it even when the 3rd person had spawned something like an RVI Agora. This doesn't ALWAYS happen, but it happened pretty consistently (more than 75% of the time). For some reason one guy we were playing with never caused any lag/framerate stutter even though we could see his vehicle, but anyone else who joined (like 5 different people) all gave us the same lag/stutter as soon as they spawned a vehicle. It happens regardless of terrain or vehicle.