Broken X window system on RHEL 6.5

Latest response

It's a messy sequence of events. It started with a working RHEL 6 system and a working Gnome GUI. The customer installed Teamviewer 9, then realized they only have a license for Teamviewer 8. So they removed Teamviewer 9 and started installing Teamviewer 8. Somewhere in the install, the system hung. They logged out of the GUI and after that, no more graphical logins.

I entered the picture and had them try a yum update. No good. I yum removed and reinstalled a bunch of stuff - most recently:

yum groupremove "General Purpose Desktop" "X Window System" "Legacy X Window System compatibility"

reboot

yum groupinstall "General Purpose Desktop" "X Window System" "Legacy X Window System compatibility"
yum groupinstall Desktop

reboot

The console just hangs until I do "init 3" from another window. It keeps a log when it starts. Here is a an extract from when the failure occurs:

[root@nfs2013 log]# more Xorg.0.log
[ 1129.167]
X.Org X Server 1.13.0
Release Date: 2012-09-05
[ 1129.169] X Protocol Version 11, Revision 0
[ 1129.171] Build Operating System: x86-027 2.6.18-371.el5
[ 1129.174] Current Operating System: Linux nfs2013.lme.local 2.6.32-431.11.2.el6.x86_64 #1 SMP Mon Mar 3
13:32:45 EST 2014 x86_64
[ 1129.175] Kernel command line: ro root=/dev/mapper/vg_system-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_
MD SYSFONT=latarcyrheb-sun16 crashkernel=128M rd_LVM_LV=vg_system/lv_swap rd_LVM_LV=vg_system/lv_root KEY
BOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
[ 1129.183] Build Date: 18 December 2013 05:39:43PM
[ 1129.184] Build ID: xorg-x11-server 1.13.0-23.1.el6_5
[ 1129.185] Current version of pixman: 0.26.2
[ 1129.188] Before reporting problems, check https://www.redhat.com/apps/support/
to make sure that you have the latest version.
[ 1129.189] Markers: (--) probed, (**) from config file, (==) default setting,
(++) from command line, (!!) notice, (II) informational,
(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
[ 1129.197] (==) Log file: "/var/log/Xorg.0.log", Time: Mon Apr 21 17:41:32 2014
[ 1129.217] (==) Using system config directory "/usr/share/X11/xorg.conf.d"
[ 1129.222] (==) No Layout section. Using the first Screen section.
[ 1129.222] (==) No screen section available. Using defaults.
.
.
.
[ 1135.596] OPTION2 = 00000000
[ 1135.596] CRTCEXT: 10 40 00 83 00 00
[ 1135.618] (==) MGA(0): Backing store disabled
[ 1135.618] (==) MGA(0): Silken mouse enabled
[ 1135.620] (==) MGA(0): DPMS enabled
[ 1135.620] (II) MGA(0): RandR 1.2 enabled, ignore the following RandR disabled message.
[ 1135.620] (WW) MGA(0): Direct rendering disabled
[ 1135.620] (--) RandR disabled
[ 1135.641] (II) SELinux: Disabled by boolean
[ 1135.643] (II) AIGLX: Screen 0 is not DRI2 capable
[ 1135.643] (II) AIGLX: Screen 0 is not DRI capable
[ 1135.792] (II) AIGLX: Loaded and initialized swrast
[ 1135.792] (II) GLX: Initialized DRISWRAST GL provider for screen 0
[ 1135.793] (II) MGA(0): Setting screen physical size to 304 x 228
[ 1136.004] (EE) XKB: Couldn't open rules file /usr/share/X11/xkb/rules/evdev
[ 1136.004] (EE) XKB: Failed to load keymap. Loading default keymap instead.
[ 1136.004] (EE) XKB: Couldn't open rules file /usr/share/X11/xkb/rules/evdev
[ 1136.004] XKB: Failed to compile keymap
[ 1136.004] Keyboard initialization failed. This could be a missing or incorrect setup of xkeyboard-config.
[ 1136.004]
Fatal server error:
[ 1136.004] Failed to activate core devices.
[ 1136.004] (EE)
Please consult the Red Hat, Inc. support
at https://www.redhat.com/apps/support/
for help.
[ 1136.004] (EE) Please also check the log file at "/var/log/Xorg.0.log" for additional information.
[ 1136.004] (EE)
[ 1136.005] Setting DAC:
00: 00 00 00 00 55 0D 00 00 CF 08 10 00 47 40 9E 00
10: 37 21 BA 00 00 00 00 00 03 07 01 00 00 20 1F 00
20: 44 39 1D 92 00 00 00 00 00 00 00 0F 14 F1 01 40
30: 00 00 00 00 00 00 00 00 00 00 07 00 98 6E 0D 00
40: A7 00 B2 00 11 90 07 00 0C 75 07 00 09 78 00 40
OPTION = 00049120
[ 1136.006] OPTION2 = 00008000
[ 1136.006] CRTCEXT: 10 40 21 83 00 3B
[ 1136.593] Server terminated with error (1). Closing log file.
[root@nfs2013 log]#

When I try to run /usr/bin/Xorg by hand from the console, it generates a bunch of output similar to above, then exits.

Various forums and documentation tell me the "Keyboard initialization failed..." error above happens when the system runs out of disk space. If only it were so simple here. I have plenty of free space and plenty of room for more files.

[root@nfs2013 log]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_system-lv_root
899G 4.8G 849G 1% /
tmpfs 7.7G 0 7.7G 0% /dev/shm
/dev/sda2 485M 93M 367M 21% /boot
/dev/sda1 200M 260K 200M 1% /boot/efi
/dev/mapper/vg_data-lv_data
2.7T 1.3T 1.3T 51% /data
[root@nfs2013 log]#
[root@nfs2013 log]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/vg_system-lv_root
59826176 155180 59670996 1% /
tmpfs 2018291 1 2018290 1% /dev/shm
/dev/sda2 128016 36 127980 1% /boot
/dev/sda1 0 0 0 - /boot/efi
/dev/mapper/vg_data-lv_data
183107584 29206245 153901339 16% /data
[root@nfs2013 log]#

Why not log a service call? This customer only has self support.

This is driving me nuts. Do I need to yum install something more? Is there a config file messed up? How do I dig into that X Window Xorg program to see why it's blowing up?

The GUI worked just fine on this system until last week. It was connected to a KVM switch with some issues. Right now, it's connected to its own monitor, keyboard, and mouse and disconnected from the KVM switch. I hooked everything into front USB and VGA ports. The hardware is an IBM x3630 M4.

thanks

  • Greg Scott

Responses

Hey Greg - I would check a few things (and these are quite possibly a stretch...)
make sure that /tmp and /var/tmp is rwxrwxrwxt.
Also - If they are using SElinux, I would try temp disabling that - or.. I would restorecon -RFvv /tmp

This seems like something is blocking access to either a file, or a device-file.

And I SERIOUSLY doubt this would help, but you could check out that rpm - xkeyboard-config

Thanks James. I looked at some of that already. /tmp and /var/tmp are OK.

[root@nfs2013 ~]# ls -al /var | grep tmp
drwxrwxrwt. 2 root root 4096 Apr 21 17:52 tmp
[root@nfs2013 ~]# ls -al / | grep tmp
drwxrwxrwt. 18 root root 4096 Apr 22 03:28 tmp
[root@nfs2013 ~]#

Oh wow - SELINUX is enforcing!!!

I always change this to permissive. I wonder if that Teamviewer installation messed with a label somewhere?

SELinux is not the culprit. Changed to permissive and no change in behavior. I see 2 more possibilities. 1. I see a bunch of log mesages about keyboard and mouse connected and disconnected. I wonder if IBM got cheap with those front USB slots? Doesn't sound like IBM but easy to test. Or 2, what if that Teamviewer app did something to a keyboard driver file?

Another thought, I have read that teamviewer may change one's grub. See if your grub.conf file has been changed recently. Also, try copying your /etc/X11/xorg.conf to /etc/X11/xorg.conf.NOTUSED them remove your xorg.conf (AFTER BACKING IT UP) temporarily and go to init 3 then to init 5. (In a Red Hat class I went to once, they said moving the xorg.conf file out of the way sometimes helps, but that one's mileage may vary, not meant as long-term, just for diagnosing.) Make note of the selinux context of the xorg.conf file. The cp for xorg.conf should retain your selinux context on xorg.conf, make a note of the context prior to the copy and restorecon later if needed if/when you place it back.

Good luck

Is Teamviewer supported on RHEL 6? Have you pinged the manufacturer? I would assume that although you have self-support on RHEL, Teamviewer is a paid third-party ISV app that has support?

Running things down - there is no xorg.conf or Xorg.conf anywhere on that system. I see this every 50 or seconds in /var/log/messages:

Apr 22 16:11:48 nfs2013 kernel: usb 2-1.4: USB disconnect, device number 123
Apr 22 16:11:50 nfs2013 kernel: usb 2-1.4: new low speed USB device number 124 using ehci_hcd
Apr 22 16:11:50 nfs2013 kernel: usb 2-1.4: New USB device found, idVendor=093a, idProduct=2510
Apr 22 16:11:50 nfs2013 kernel: usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Apr 22 16:11:50 nfs2013 kernel: usb 2-1.4: Product: USB OPTICAL MOUSE
Apr 22 16:11:50 nfs2013 kernel: usb 2-1.4: Manufacturer: PIXART
Apr 22 16:11:50 nfs2013 kernel: usb 2-1.4: configuration #1 chosen from 1 choice
Apr 22 16:11:50 nfs2013 kernel: input: PIXART USB OPTICAL MOUSE as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.0/input/input246
Apr 22 16:11:50 nfs2013 kernel: generic-usb 0003:093A:2510.00F5: input,hidraw4: USB HID v1.11 Mouse [PIXART USB OPTICAL MOUSE] on usb-0000:00:1d.0-1.4/input0

  • Greg

Connecting the USB mouse and keyboard in the rear USB slots makes no difference.

When I do startx by hand or if I try /usr/bin/Xorg from the console I see this in /var/log/messages:

Apr 22 16:20:04 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:20:04 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:20:04 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:20:04 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:20:04 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:20:04 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:20:04 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:20:04 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:20:04 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents

No idea if this is normal or not.

When I do "init 5" I see this in /var/log/messages:

Apr 22 16:22:16 nfs2013 auditd[2607]: The audit daemon is exiting.
Apr 22 16:22:16 nfs2013 kernel: type=1305 audit(1398201736.742:1135): audit_pid=0 old=2607 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
Apr 22 16:22:16 nfs2013 kernel: type=1305 audit(1398201736.908:1136): audit_enabled=0 old=1 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditctl_t:s0 res=1
Apr 22 16:22:17 nfs2013 init: Failed to spawn prefdm main process: unable to execute: No such file or directory
Apr 22 16:22:17 nfs2013 init: start-ttys main process (19814) terminated with status 1

grub.conf looks normal. Both grub.conf and grub.conf.rpmsave are identical and dated Feb. 13, 2013:

[root@nfs2013 ~]# ls -al /etc | grep grub
lrwxrwxrwx. 1 root root 32 Feb 14 2013 grub.conf -> ../boot/efi/EFI/redhat/grub.conf
lrwxrwxrwx. 1 root root 33 Feb 14 2013 grub.conf.rpmsave -> ..//boot/efi/EFI/redhat/grub.conf
[root@nfs2013 ~]# more /etc/grub.conf

grub.conf generated by anaconda

#

Note that you do not have to rerun grub after making changes to this file

NOTICE: You have a /boot partition. This means that

all kernel and initrd paths are relative to /boot/, eg.

root (hd0,1)

kernel /vmlinuz-version ro root=/dev/mapper/vg_system-lv_root

initrd /initrd-[generic-]version.img

boot=/dev/sda1

device (hd0) HD(1,800,64000,d30e0b9d-e792-40a1-8da6-933b329db3a3)
default=0
timeout=5
splashimage=(hd0,1)/grub/splash.xpm.gz
hiddenmenu
title Red Hat Enterprise Linux Server (2.6.32-431.11.2.el6.x86_64)
root (hd0,1)
kernel /vmlinuz-2.6.32-431.11.2.el6.x86_64 ro root=/dev/mapper/vg_system-lv_root rd_NO_LUKS LANG=en
_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M rd_LVM_LV=vg_system/lv_swap rd_LVM_LV=vg_syst
em/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
initrd /initramfs-2.6.32-431.11.2.el6.x86_64.img
title Red Hat Enterprise Linux Server (2.6.32-279.22.1.el6.x86_64)
root (hd0,1)
kernel /vmlinuz-2.6.32-279.22.1.el6.x86_64 ro root=/dev/mapper/vg_system-lv_root rd_NO_LUKS LANG=en
_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M rd_LVM_LV=vg_system/lv_swap rd_LVM_LV=vg_syst
em/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
initrd /initramfs-2.6.32-279.22.1.el6.x86_64.img
title Red Hat Enterprise Linux (2.6.32-279.el6.x86_64)
root (hd0,1)
kernel /vmlinuz-2.6.32-279.el6.x86_64 ro root=/dev/mapper/vg_system-lv_root rd_NO_LUKS LANG=en_US.U
TF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M rd_LVM_LV=vg_system/lv_swap rd_LVM_LV=vg_system/lv
_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
initrd /initramfs-2.6.32-279.el6.x86_64.img
[root@nfs2013 ~]#

I may have fixed my prefdm problem. When I did all my reinstalls, I wanted to make sure it was a completely clean install. So I renamed /etc/X11 to /etc/X11-messedup and then reinstalled all the GUI stuff I could find. Turns out, X11-messedup had a copy of an executable named prefdm. I copied that to /etc/X11 and the prefdm error went away with init 5. But I still have a bunch more to go:

Apr 22 16:54:05 nfs2013 kernel: type=1305 audit(1398203645.459:1193): audit_pid=0 old=19899 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
Apr 22 16:54:05 nfs2013 auditd[19899]: The audit daemon is exiting.
Apr 22 16:54:05 nfs2013 kernel: type=1305 audit(1398203645.606:1194): audit_enabled=0 old=1 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditctl_t:s0 res=1
Apr 22 16:54:06 nfs2013 init: start-ttys main process (22148) terminated with status 1
Apr 22 16:54:06 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:06 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:06 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:06 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:06 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:06 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:06 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:06 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:06 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:08 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:08 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:08 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:08 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:08 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:08 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:08 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:08 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:08 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:10 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:10 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:10 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:10 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:10 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:10 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:10 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:10 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:10 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:11 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:11 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:11 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:11 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:11 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:11 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:11 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:11 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:11 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:13 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:13 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:13 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:13 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:13 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:13 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:13 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:13 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:13 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:14 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:14 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:14 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:14 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:14 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:14 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:14 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:14 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:14 nfs2013 kernel: pci 0000:05:00.0: Invalid ROM contents
Apr 22 16:54:27 nfs2013 kernel: usb 2-1.3: USB disconnect, device number 47
Apr 22 16:54:29 nfs2013 kernel: usb 2-1.3: new low speed USB device number 48 using ehci_hcd
Apr 22 16:54:29 nfs2013 kernel: usb 2-1.3: New USB device found, idVendor=093a, idProduct=2510
Apr 22 16:54:29 nfs2013 kernel: usb 2-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Apr 22 16:54:29 nfs2013 kernel: usb 2-1.3: Product: USB OPTICAL MOUSE
Apr 22 16:54:29 nfs2013 kernel: usb 2-1.3: Manufacturer: PIXART
Apr 22 16:54:29 nfs2013 kernel: usb 2-1.3: configuration #1 chosen from 1 choice
Apr 22 16:54:29 nfs2013 kernel: input: PIXART USB OPTICAL MOUSE as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.3/2-1.3:1.0/input/input301
Apr 22 16:54:29 nfs2013 kernel: generic-usb 0003:093A:2510.012C: input,hidraw2: USB HID v1.11 Mouse [PIXART USB OPTICAL MOUSE] on usb-0000:00:1d.0-1.3/input0
^C
[root@nfs2013 ~]#

I wonder why this happens in /var/log/messages now when I do init 5?

Apr 22 17:12:22 nfs2013 init: start-ttys main process (23729) terminated with status 1

I should close the loop on this. And put in a plug for Red Hat Support, which helped me fix it. I logged a case against my own stuff instead of the customer and I'll push the customer to go to Standard Support instead of Self Support with the next renewal cycle coming up in May. The problem turned out to be a bunch of missing RPMs. I have no clue how yum groupremove and yum groupinstall did not put everything back. But we did rpm -Va and found a bazillion missing files. Going through each missing file with yum whatprovides, I did yum reinstall for around 10 or so RPMs. And after that, it all worked. Go figure.

It's possible somebody individually deleted a bunch of files or directories. It would be nice to know why they disappeared and why yum groupremove and yum groupinstall didn't put them back. But the system is back up and running now.

  • Greg

Glad you got this one resolved, Greg. Thanks for sharing the solution here, and thanks for the plug too. ;)

Thanks, it reminded me to check and install the X11 drivers rpm for a minimal desktop setup I'm setting up.