RHEL 6 on VMware /dev/random entropy issue
Hi
I was wondering if anyone has had issues with the /dev/random RNG device being extremely slow on RHEL 6.5 when running as an VMware guest? When ssh-ing between servers the ssh login process hangs for many 10s of seconds (sometimes minutes) and key generation, using ssh-keygen, also hangs for ages. With some debugging I have narrowed the issue down to the processes waiting for /dev/random. Linking /dev/random (as per http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1036980) to /dev/urandom makes the processes run at the expected speeds.
I have an open case regarding the issue and I suggested that the link might be a good work-around (based on research into differences between /dev/random and urandom [e.g. http://www.2uo.de/myths-about-urandom/] but RH support are suggesting a hardware number generator. I've never used a hardware number generator but I suspect, in a non-vm environment, it is a straightforward to setup. I am, however, wondering if anyone has had experience in using one in a VMware environment with many guests per physical host? Can one hardware number generator feed multiple VMs?
(FYI: The RHEL 6 build in question has had the cc-eal4-config-rhel62 configuration applied which includes setting "SSH_USE_STRONG_RNG=12" in /etc/sysconfig/sshd, among other things!)
Thanks in advance,
Aidan
Responses
Guru
6460 points
FWIW: the /dev/random being "slow" thing isn't a RedHat-only issue. I've admin a number of different commercial and OSS based systems and use of /dev/random is always slow in comparison to /dev/urandom (blocking versus non-blocking entropy service).
Can't really speak to how well ESX bubbles up its hardware-based entropy subsystems. That said...
Typically, if /dev/random proves too slow and /dev/urandom is deemed to suffer from insufficient randomness, you'd look at a solution like PRNG. However, PRNG is best-suited to graphical (physical) workstations rather than headless servers. You might want to look at the "haveged" service as an alternative: you can get it via EPEL.
Red Hat
Guru
11969 points
That 2uo page you linked to is fantastic. Finally someone who collects all the myths about randomness and debunks them.
I have done a bit of research into this, though I have little actual configuration experience.
Older Linux kernels used to read from many IRQ sources like network interfaces, following the theory that with all the traffic happening on a LAN which a system sees, the NIC's behaviour at supplying that traffic would be unpredictable. At some point in the past, someone decided it was possible to game that if you could control traffic on the LAN. Nowadays the only sources of entropy are human interfaces like keyboards and mice, and actual hardware entropy generators.
VMWare doesn't provide hardware entropy to guests at all, which is why /dev/random starves.
My first advice would be to actually test /dev/urandom. In my own tests, inside a KVM VM with no source of hardware randomness, /dev/urandom could supply literally megabytes of randomness before numbers became predictable enough to fail a FIPS-140 test.
You can test any entropy source for FIPS compliance with rngtest which is supplied in the rng-tools package. See man rngtest for the correct syntax, many examples are also available which a quick web search.
If you control the software in question, one option is to use the small amount of randomness which /dev/random supplies to seed your own PRNG algorithm inside your application. As the page you linked to says, even 256 bytes (2048 bits) should be a large enough seed.
Like Tom said, you can run a PRNG entropy daemon like haveged. This reads randomness from tiny fluctuations in the processor's TSC timer (also emulated in VMWare) to seed random numbers. The theory is explained in-depth on the haveged website.
It appears you've read about hardware randomness generators, usually these are little USB dongles which collect some sort of environmental thermal/photoelectric/quantum noise from around them and use it to seed a hardware RNG. Wikipedia has a comparison of some. As far as I understand, one device can only feed one system. This would imply you'd need one hardware dongle per VM (and a lot of USB ports!) which leads into our next option.
You can also use an "entropy broker". This is a daemon which runs on a true source of hardware randomness, and uses a network client/server can send that randomness to multiple systems.
IIUC you could have a system (physical or virtual) with a USB RNG running a broker which supplies hardware randomness to several other machines. I guess your entropy requirements would dictate if/when you'd starve one RNG enough to require more generators and brokers.
Looping back over all these options, the first thing I would do is test /dev/urandom with rngtest. Test it with the amount of entropy you actually need and see if it stays sufficiently random over time. In my (admittedly limited) experience, you'll be able to get a lot of unpredicable data before resorting to any of the above methods.
As a final note, RHEL 6.6 beta is testing virtio-rng, a paravirtualized hardware random number generator which feeds KVM hypervisor entropy to KVM guests. Check the Release Notes after 6.6 comes out to see if this was included in the GA release, and whether it's Tech Preview or fully supported.
Update: RHEL 6.6 and later support virtio-rng, described at Does the Red Hat Enterprise Linux KVM Hypervisor support the VirtIO Random Number Generator?.
Red Hat
Guru
3878 points
That 2uo article was new to me. Masterfully-done. Wow.
I gave my own shot at writing something similar a few years ago when I started at Red Hat: Entropy & the Linux kernel: /dev/random versus /dev/urandom
Newbie
5 points
Here's redhat's recommendations for devices that are unable to maintain a sufficient entropy pool size. How to increase the entropy pool without using a keyboard or mouse
Guru
6888 points
Can confirm this is neither a VMware Issue nor a RHEL issue (specific to them). I ran in to the same issue with physical nodes. Fortunately we were able to redirect our application to use /dev/urandom. There are some implications that need to be reviewed, however.
Guru
6688 points
I haven't seen this issue for a while (have seen it on VM webservers before). I find now, a lot of places have moved the SSL termination off to hardware load balancers in front of the webservers, which is potentially another option if you are having this issue specifically with SSL (not a Red Hat solution I know!).
Guru
4182 points
There is a recent blog article that covers RHEL 7:
Red Hat Enterprise Linux Virtual Machines: Access to Random Numbers Made Easy
Community Member
47 points
Why is the "best response" to a question about entropy on VMware a link to a page that discusses how to resolve the problem on KVM? In my opinion the best responses are Tom's and Jamie's, which actually address the original question.