How to enable serial console logs in ppc64le system

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux
  • ppc64le systems

Issue

Serial console logs are necessary to troubleshoot specific issues. How are they enabled in Power system?

Resolution

Find the name of the serial console from the dmesg output. Execute dmesg on server or reference a sosreport:

198 [ 0.000141] Console: colour dummy device 80x25
199 [ 0.000159] printk: console [hvc0] enabled            <<<<

Primary kernel
Check if the parameter was enabled on the kernel command line:
# grep console /proc/cmdline
look for a reference to the named printk serial device above:

[root@test]# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-4.18.0-553.50.1.el8_10.ppc64le root=/dev/mapper/test ro rd.lvm.lv=rhel_ibm-p8-pvm-09-guest-15/root rd.lvm.lv=rhel_ibm-p8-pvm-09-guest-15/swap crashkernel=256M

If it's not present then add the parameter console=hvc0 or the specific serial device by editing the kernel command line. Here are references to edit it:

Kdump kernel

  1. Add the next parameter in /etc/sysconfig/kdump file, in the line that begins with KDUMP_COMMANDLINE_APPEND so that kdump kernel logs are visible on serial console.
    console=hvc0,115200n8
KDUMP_COMMANDLINE_APPEND="irqpoll maxcpus=1 noirqdistrib reset_devices cgroup_disable=memory numa=off udev.children-max=2 ehea.use_mcs=0 panic=10 rootflags=nofail kvm_cma_resv_ratio=0 transparent_hugepage=never novmcoredd console=hvc0,115200n8"
  1. Restart the kdump service:
 # touch /etc/kdump.conf
 # systemctl restart kdump.service

  1. Engage IBM if needing assistance capturing serial console output at the hypervisor level.

  2. After the above changes, schedule a down window, perform the kdump testing and provide serial console logs with fresh sosreport in case the vmcore does not get collected under /var/crash timestamp directory.

    During a downtime, test sending the "System Rest Interrupt" to the guest. It should experience a kernel panic and reboot. Find information about it in: How to Manually Initiate a Vmcore Dump from a Power System on LPAR

  3. If the test is successful you will see similar messages to:

  • If the kdump service is working:
[   22.303875] systemd[1]: Starting Kdump Vmcore Save Service...
[   22.523878] kdump[406]: Kdump is using the default log level(3).
[   22.739586] kdump[428]: saving to /sysroot//var/crash/127.0.0.1-2025-04-15-13:55:15/
[   22.779363] kdump[432]: saving vmcore-dmesg.txt to /sysroot//var/crash/127.0.0.1-2025-04-15-13:55:15/
[   22.828389] kdump[438]: saving vmcore-dmesg.txt complete
[   22.836790] kdump[440]: saving vmcore
Copying data                                      : [100.0 %] /           eta: 0s
[   26.797468] kdump.sh[441]: The dumpfile is saved to /sysroot//var/crash/127.0.0.1-2025-04-15-13:55:15/vmcore-incomplete.
[   26.797861] kdump.sh[441]: makedumpfile Completed.
[   27.746555] kdump[445]: saving vmcore complete
[   27.754849] kdump[447]: saving the /run/initramfs/kexec-dmesg.log to /sysroot//var/crash/127.0.0.1-2025-04-15-13:55:15/
[   27.829518] kdump[453]: Executing final action systemctl reboot -f
[   27.880810] systemd[1]: Shutting down.
[   28.066134] printk: systemd-shutdow: 20 output lines suppressed due to ratelimiting
[   28.769005] systemd-shutdown[1]: Syncing filesystems and block devices.
[   28.771318] systemd-shutdown[1]: Sending SIGTERM to remaining processes...
[   28.790188] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[   28.794269] systemd-shutdown[1]: Unmounting file systems.
[   28.796275] [457]: Remounting '/sysroot' read-only in with options 'attr2,inode64,logbufs=8,logbsize=32k,noquota,x-systemd.before=initrd-fs.target'.
[   28.798628] xfs: Unknown parameter 'x-systemd.before'
[   28.798988] [457]: Failed to remount '/sysroot' read-only: Invalid argument
[   28.799836] [458]: Unmounting '/sysroot'.
[   28.801832] XFS (dm-0): Unmounting Filesystem
[   28.805489] [459]: Remounting '/' read-only in with options 'lowerdir=/squash/root,upperdir=/squash/overlay/upper,workdir=/squash/overlay/work/'.
[   28.810832] [460]: Unmounting '/squash/root'.
[   28.811672] [461]: Unmounting '/squash'.
[   28.813114] systemd-shutdown[1]: All filesystems unmounted.
[   28.813128] systemd-shutdown[1]: Deactivating swaps.
[   28.813168] systemd-shutdown[1]: All swaps deactivated.
[   28.813182] systemd-shutdown[1]: Detaching loop devices.
[   28.813995] systemd-shutdown[1]: Detaching loopback /dev/loop0.
[   28.814034] systemd-shutdown[1]: All loop devices detached.
[   28.850865] reboot: Restarting system
  • If the kdump service is not working, open a support case for Red Hat support to review.

Root Cause

In power systems it may be necessary to enable serial console logs to troubleshoot several problems. There can be various reasons why a vmcore is not produced which why reviewing the kdump messages presented in the logs to understand for the next steps.

Serial console redirection requires three parts:

  1. The setting enabled in the BIOS.
  2. The serial console configured with the correct settings to receive the input.
  3. The setting on the grub command line for the OS to forward the text.

Diagnostic Steps

Check in the sosreport messages which console is available or execute dmesg on server:

198 [ 0.000141] Console: colour dummy device 80x25
199 [ 0.000159] printk: console [hvc0] enabled

In this example hvc0 is available.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments