Chapter 4. Troubleshooting CDK

4.1. Overview

This section contains solutions to common problems that you might encounter while setting up and using CDK.

4.2. Troubleshooting Getting Started

4.2.1. Overview

This section contains solutions to common problems that you might encounter while installing and configuring CDK.

4.2.2. CDK startup check failed

While CDK starts, it runs several startup checks to make sure that the CDK VM and the OpenShift Cluster are able to start without any issues. If any configuration is incorrect or missing, the startup checks fail and CDK does not start.

The following sections describe the different startup checks.

4.2.2.1. Driver plug-in configuration

One of the startup checks verifies that the relevant driver plug-in is configured correctly. If this startup check fails, review the set up your virtualization environment topic and configure the driver.

If you want to force CDK to start despite a failing driver plug-in check, you can instruct CDK to treat these errors as warnings:

  • For KVM/libvirt on Linux, run the following command:

    $ minishift config set warn-check-kvm-driver true
  • For xhyve on macOS, run the following command:

    $ minishift config set warn-check-xhyve-driver true
  • For Hyper-V on Windows, run the following command:

    C:\> minishift.exe config set warn-check-hyperv-driver true

4.2.2.2. Persistent storage volume configuration and usage

CDK checks whether the persistent storage volume is mounted and that enough disk space is available. If the persistent storage volume, for example, uses more than 95% of the available disk space, CDK will not start.

If you want to recover the data, you can skip this test and start CDK to access the persistent volume:

$ minishift config set skip-check-storage-usage true

4.2.2.3. External network connectivity

After the CDK VM starts, it runs several network checks to verify whether external connectivity is possible from within the CDK VM.

By default, network checks are configured to treat any errors as warnings, because of the diversity of the development environments. You can configure the network checks to optimize them for your environment.

For example, one of the network checks pings an external host. You can change the host by running the following command:

$ minishift config set check-network-ping-host <host-IP-address>

Replace <host-IP-address> with the address of your internal DNS server, proxy host, or an external host that you can reach from your machine.

Because proxy connectivity might be problematic, you can run a check that tries to retrieve an external URL. You can configure the URL by running:

$ minishift config set check-network-http-host <URL>

4.2.3. OpenShift web console does not work with older versions of Safari

minishift console does not work on older versions of Safari web browser such as version 10.1.2 (12603.3.8). Attempting to access the web console results in the following error:

Error unable to load details about the server

Retry after updating it to the latest version or use Firefox or Chrome browser for this. Version 11.0.3 (13604.5.6) has been tested and works with OpenShift web console. You can use minishift console --url to get the web console URL.

4.3. Troubleshooting Driver Plug-ins

4.3.1. Overview

This section contains solutions to common problems that you might encounter while configuring the driver plug-ins for CDK.

4.3.2. KVM/libvirt

4.3.2.1. Undefining virsh snapshots fail

If you use virsh on KVM/libvirt to create snapshots in your development workflow then use minishift delete to delete the snapshots along with the VM, you might encounter the following error:

$ minishift delete
Deleting the Minishift VM...
Error deleting the VM:  [Code-55] [Domain-10] Requested operation is not valid: cannot delete inactive domain with 4 snapshots

Cause: The snapshots are stored in ~/.minishift/machines, but the definitions are stored in /var/lib/libvirt/qemu/snapshot/minishift.

Workaround: To delete the snapshots, you need to perform the following steps as root.

  1. Delete the definitions:

    # virsh snapshot-delete --metadata minishift <snapshot-name>
  2. Undefine the CDK domain:

    # virsh undefine minishift

    You can now run minishift delete to delete the VM and restart CDK.

Note

If these steps do not resolve the issue, you can also use the following command to delete the snapshots:

$ rm -rf ~/.minishift/machines

It is recommended to avoid using metadata when you create snapshots. To ensure this, you can specify the --no-metadata flag. For example:

# virsh snapshot-create-as --domain vm1 overlay1 --diskspec vda,file=/export/overlay1.qcow2 --disk-only --atomic --no-metadata

4.3.2.2. Error creating new host: dial tcp: missing address

The problem is likely that the libvirtd service is not running. You can check this with the following command:

$ systemctl status libvirtd

If libvirtd is not running, start it and enable it to start on boot:

$ systemctl start libvirtd
$ systemctl enable libvirtd

4.3.2.3. Failed to connect socket to '/var/run/libvirt/virtlogd-sock'

The problem is likely that the virtlogd service is not running. You can check this with the following command:

$ systemctl status virtlogd

If virtlogd is not running, start it and enable it to start on boot:

$ systemctl start virtlogd
$ systemctl enable virtlogd

4.3.2.4. Domain 'minishift' already exists…​

If you try minishift start and this error appears, ensure that you use minishift delete to delete the VMs that you created earlier. However, if this fails and you want to completely clean up CDK and start fresh, do the following:

  1. As root, check if any existing CDK VMs are running:

    # virsh list --all
  2. If any CDK VM is running, stop it:

    # virsh destroy minishift
  3. Delete the VM:

    # virsh undefine minishift
  4. As your regular user, delete the ~/.minishift/machines directory:

    $ rm -rf ~/.minishift/machines

In case all of this fails, you might want to uninstall CDK and do a fresh install of CDK.

4.3.3. xhyve

4.3.3.1. Could not create vmnet interface

The problem is likely that the xhyve driver is not able to clean up vmnet when a VM is removed. vmnet.framework determines the IP address based on the following files:

  • /var/db/dhcpd_leases
  • /Library/Preferences/SystemConfiguration/com.apple.vmnet.plist

Reset the CDK-specific IP database, ensure that you remove the minishift entry section from the dhcpd_leases file, and reboot your system.

{
  ip_address=192.168.64.2
  hw_address=1,2:51:8:22:87:a6
  identifier=1,2:51:8:22:87:a6
  lease=0x585e6e70
  name=minishift
}
Note

You can completely reset the IP database by removing the files manually, but this is very risky.

4.3.4. VirtualBox

4.3.4.1. Error machine does not exist

If you use Windows, ensure that you set the --vm-driver virtualbox flag in the minishift start command. Alternatively, the problem might be an outdated version of VirtualBox.

To avoid this issue, it is recommended to use VirtualBox 5.1.12 or later.

4.3.5. Hyper-V

4.3.5.1. Hyper-V commands must be run as an Administrator

If you run CDK with Hyper-V on Windows as a normal user or as a user with Administrator privileges, you might encounter the following error:

Error starting the VM: Error creating the VM. Error with pre-create check: "Hyper-V commands must be run as an Administrator".

Workaround: You can either add yourself to the Hyper-V Administrators group, which is recommended, or run the shell in an elevated mode.

If you are using PowerShell, you can add yourself to the Hyper-V Administrators group as follows:

  1. As an administrator, run the following command:

    PS> ([adsi]”WinNT://./Hyper-V Administrators,group”).Add(“WinNT://$env:UserDomain/$env:Username,user”)
  2. Log out and log back in for the change to take effect.

You can also use the GUI to add yourself to the Hyper-V Administrators group as follows:

  1. Click the Start button and select Computer Management.
  2. In the Computer Management window, select Local Users And Groups, then double-click Groups.
  3. Double-click the Hyper-V Administrators group, the Hyper-V Administrators Properties dialog box is displayed.
  4. Add your account to the Hyper-V Administrators group, log off, then log in for the change to take effect.

Now you can run the Hyper-V commands as a normal user.

For more options for Hyper-V see creating Hyper-V administrators local group.

4.3.5.2. CDK running with Hyper-V fails when connected to OpenVPN

If you try to use CDK with Hyper-V using an external virtual switch while you are connected to a VPN such as OpenVPN, CDK might fail to provision the VM.

Cause: Hyper-V networking might not route the network traffic in both directions properly when connected to a VPN.

Workaround: Disconnect from the VPN and try again after stopping the VM from the Hyper-V manager.

4.4. Troubleshooting Miscellaneous

4.4.1. Overview

This section contains solutions to common problems that you might encounter while using various components of CDK.

4.4.2. The root filesystem of the CDK VM exceeds overlay size

Installing additional packages or copying large files to the root filesystem of the CDK VM might exceed the allocated overlay size and lock the CDK VM.

Cause: The CDK VM root filesystem contains core packages that are configured to optimize running the CDK VM and containers. The available storage on the root filesystem is determined by the overlay size, which is smaller than the total available storage.

Workaround: Avoid installing packages or storing large files in the root filesystem of the CDK VM. Instead, you can create a sub-directory in the /mnt/sda1/ persistent storage volume or define and mount host folders that can share storage space between the host and the CDK VM.

If you want to perform development tasks inside the CDK VM, it is recommended that you use containers, which are stored in persistent storage volumes, and reuse the CDK Docker daemon.

4.4.3. Special characters cause passwords to fail

Depending on your operating system and shell environment, certain special characters can trigger variable interpolation and therefore cause passwords to fail.

Workaround: When creating and entering passwords, wrap the string with single quotes in the following format: '<password>'

4.4.4. X.509 certificate is valid for 10.0.2.15, 127.0.0.1, 172.17.0.1, 172.30.0.1, 192.168.99.100, not 192.168.99.101

Starting a stopped CDK VM can produce the following X.509 certificate error:

$ minishift start
-- Checking if requested hypervisor 'kvm' is supported on this platform ... OK
-- Checking the ISO URL ... OK
-- Starting local OpenShift cluster using 'kvm' hypervisor ...
-- Starting Minishift VM ......................... OK
[...]
FAIL
   Error: cannot access master readiness URL https://192.168.99.101:8443/healthz/ready
   Details:
     No log available from "origin" container

   Caused By:
     Error: Get https://192.168.99.101:8443/healthz/ready: x509: certificate is valid for 10.0.2.15, 127.0.0.1, 172.17.0.1, 172.30.0.1, 192.168.99.100, not 192.168.99.101
Error during 'cluster up' execution: Error starting the cluster.

The reason for the above error is that OpenShift cluster certificates contain the IP of the CDK VM. The certificates are generated only when the CDK VM is freshly started. After restart, the CDK VM might be assigned a new IP address. If this happens, the certificate becomes invalid.

Workaround: Delete the existing CDK VM and start again.

$ minishift delete --force
$ minishift start