Why do I see strange spikes in latency within RHEL guests running in VMware environments?

Issue

The observation is that some installations of Red Hat Enterprise Linux guests in a VMware environment experience strange latency spikes for no apparent reason. An observation of this in practice would look something like this:

$ ping guest1 -i 0.1 | grep -v "time=0."
PING guest1.fqdn (192.168.0.10) 56(84) bytes of data.
64 bytes from guest1.fqdn (192.168.0.10): icmp_seq=33 ttl=64 time=6.89 ms
64 bytes from guest1.fqdn (192.168.0.10): icmp_seq=34 ttl=64 time=5.71 ms
64 bytes from guest1.fqdn (192.168.0.10): icmp_seq=70 ttl=64 time=2.37 ms
64 bytes from guest1.fqdn (192.168.0.10): icmp_seq=102 ttl=64 time=6.58 ms
64 bytes from guest1.fqdn (192.168.0.10): icmp_seq=117 ttl=64 time=2.55 ms
64 bytes from guest1.fqdn (192.168.0.10): icmp_seq=135 ttl=64 time=6.33 ms
64 bytes from guest1.fqdn (192.168.0.10): icmp_seq=171 ttl=64 time=8.19 ms
64 bytes from guest1.fqdn (192.168.0.10): icmp_seq=172 ttl=64 time=7.74 ms

While on a guest that works properly, the following is observed:

$ ping guest2 -i 0.1 | grep -v "time=0."
PING guest2.fqdn (192.168.1.10) 56(84) bytes of data.