At best, the overhead from virtual memory support presents a minimal additional load to a well-configured system:
RAM -- Sufficient RAM for all working sets with enough left over to handle any page faults
Disk -- Because of the limited page fault activity, disk I/O bandwidth would be minimally impacted
CPU -- The majority of CPU cycles are dedicated to actually running applications, instead of running the operating system's memory management code
From this, the overall point to keep in mind is that the performance impact of virtual memory is minimal when it is used as little as possible. This means the primary determinant of good virtual memory subsystem performance is having enough RAM.
Next in line (but much lower in relative importance) are sufficient disk I/O and CPU capacity. However, keep in mind that these resources only help the system performance degrade more gracefully from heavy faulting and swapping; they do little to help the virtual memory subsystem performance (although they obviously can play a major role in overall system performance).