Chapter 4. Troubleshooting CDK
This section contains solutions to common problems that you might encounter while setting up and using CDK.
4.1. Troubleshooting Getting Started
This section contains solutions to common problems that you might encounter while installing and configuring CDK.
4.1.2. CDK startup check failed
While CDK starts, it runs several startup checks to make sure that the CDK VM and the OpenShift Cluster are able to start without any issues. If any configuration is incorrect or missing, the startup checks fail and CDK does not start.
You can skip the startup checks by executing the following command:
$ minishift config set skip-startup-checks true
The following sections describe the different startup checks.
184.108.40.206. Driver plug-in configuration
One of the startup checks verifies that the relevant driver plug-in is configured correctly. If this startup check fails, review the Setting Up the Virtualization Environment topic and configure the appropriate driver.
If you want to force CDK to start despite a failing driver plug-in check, you can instruct CDK to treat these errors as warnings:
For KVM/libvirt on Linux, run the following command:
$ minishift config set warn-check-kvm-driver true
For hyperkit on macOS, run the following command:
$ minishift config set warn-check-hyperkit-driver true
For Hyper-V on Windows, run the following command:
C:\> minishift.exe config set warn-check-hyperv-driver true
220.127.116.11. Persistent storage volume configuration and usage
CDK checks whether the persistent storage volume is mounted and that enough disk space is available. If the persistent storage volume, for example, uses more than 95% of the available disk space, CDK will not start.
If you want to recover the data, you can skip this test and start CDK to access the persistent volume:
$ minishift config set skip-check-storage-usage true
18.104.22.168. External network connectivity
After the CDK VM starts, it runs several network checks to verify whether external connectivity is possible from within the CDK VM.
By default, network checks are configured to treat any errors as warnings, because of the diversity of the development environments. You can configure the network checks to optimize them for your environment.
For example, one of the network checks pings an external host. You can change the host by running the following command:
$ minishift config set check-network-ping-host <host-IP-address>
<host-IP-address> with the address of your internal DNS server, proxy host, or an external host that you can reach from your machine.
Because proxy connectivity might be problematic, you can run a check that tries to retrieve an external URL. You can configure the URL by running:
$ minishift config set check-network-http-host <URL>
4.1.3. OpenShift web console does not work with older versions of Safari
minishift console does not work on older versions of Safari web browser such as version 10.1.2 (12603.3.8). Attempting to access the web console results in the following error:
Error unable to load details about the server
Retry after updating it to the latest version or use Firefox or Chrome browser for this. Version 11.0.3 (13604.5.6) has been tested and works with OpenShift web console. You can use
minishift console --url to get the web console URL.
4.2. Troubleshooting Driver Plug-ins
This section contains solutions to common problems that you might encounter while configuring the driver plug-ins for CDK.
22.214.171.124. Undefining virsh snapshots fail
If you use
virsh on KVM/libvirt to create snapshots in your development workflow then use
minishift delete to delete the snapshots along with the VM, you might encounter the following error:
$ minishift delete Deleting the Minishift VM... Error deleting the VM: [Code-55] [Domain-10] Requested operation is not valid: cannot delete inactive domain with 4 snapshots
Cause: The snapshots are stored in ~/.minishift/machines, but the definitions are stored in /var/lib/libvirt/qemu/snapshot/minishift.
Workaround: To delete the snapshots, you need to perform the following steps as root.
Delete the definitions:
# virsh snapshot-delete --metadata minishift <snapshot-name>
Undefine the CDK domain:
# virsh undefine minishift
You can now run
minishift deleteto delete the VM and restart CDK.
If these steps do not resolve the issue, you can also use the following command to delete the snapshots:
$ rm -rf ~/.minishift/machines
It is recommended to avoid using metadata when you create snapshots. To ensure this, you can specify the
--no-metadata flag. For example:
# virsh snapshot-create-as --domain vm1 overlay1 --diskspec vda,file=/export/overlay1.qcow2 --disk-only --atomic --no-metadata
126.96.36.199. Error creating new host: dial tcp: missing address
The problem is likely that the libvirtd service is not running. You can check this with the following command:
$ systemctl status libvirtd
If libvirtd is not running, start it and enable it to start on boot:
$ systemctl start libvirtd $ systemctl enable libvirtd
188.8.131.52. Failed to connect socket to '/var/run/libvirt/virtlogd-sock'
The problem is likely that the virtlogd service is not running. You can check this with the following command:
$ systemctl status virtlogd
If virtlogd is not running, start it and enable it to start on boot:
$ systemctl start virtlogd $ systemctl enable virtlogd
184.108.40.206. Domain 'minishift' already exists…
If you try
minishift start and this error appears, ensure that you use
minishift delete to delete the VMs that you created earlier. However, if this fails and you want to completely clean up CDK and start fresh, do the following:
As root, check if any existing CDK VMs are running:
# virsh list --all
If any CDK VM is running, stop it:
# virsh destroy minishift
Delete the VM:
# virsh undefine minishift
As your regular user, delete the ~/.minishift/machines directory:
$ rm -rf ~/.minishift/machines
In case all of this fails, you might want to uninstall CDK and do a fresh install of CDK.
220.127.116.11. Error machine does not exist
If you use Windows, ensure that you set the
--vm-driver virtualbox flag in the
minishift start command. Alternatively, the problem might be an outdated version of VirtualBox.
To avoid this issue, it is recommended to use VirtualBox 5.1.12 or later.
18.104.22.168. Hyper-V commands must be run as an Administrator
If you run CDK with Hyper-V on Windows as a normal user or as a user with Administrator privileges, you might encounter the following error:
Error starting the VM: Error creating the VM. Error with pre-create check: "Hyper-V commands must be run as an Administrator".
Workaround: You can either add yourself to the Hyper-V Administrators group, which is recommended, or run the shell in an elevated mode.
If you are using PowerShell, you can add yourself to the Hyper-V Administrators group as follows:
As an administrator, run the following command:
PS> ([adsi]"WinNT://./Hyper-V Administrators,group").Add("WinNT://$env:UserDomain/$env:Username,user")
- Log out and log back in for the change to take effect.
You can also use the GUI to add yourself to the Hyper-V Administrators group as follows:
- Click the Start button and select Computer Management.
- In the Computer Management window, select Local Users And Groups, then double-click Groups.
- Double-click the Hyper-V Administrators group, the Hyper-V Administrators Properties dialog box is displayed.
- Add your account to the Hyper-V Administrators group, log off, then log in for the change to take effect.
Now you can run the Hyper-V commands as a normal user.
For more options for Hyper-V see creating Hyper-V administrators local group.
22.214.171.124. CDK running with Hyper-V fails when connected to OpenVPN
If you try to use CDK with Hyper-V using an external virtual switch while you are connected to a VPN such as OpenVPN, CDK might fail to provision the VM.
Cause: Hyper-V networking might not route the network traffic in both directions properly when connected to a VPN.
Workaround: Disconnect from the VPN and try again after stopping the VM from the Hyper-V manager.
4.3. Troubleshooting Miscellaneous
This section contains solutions to common problems that you might encounter while using various components of CDK.
4.3.2. The root filesystem of the CDK VM exceeds overlay size
Installing additional packages or copying large files to the root filesystem of the CDK VM might exceed the allocated overlay size and lock the CDK VM.
Cause: The CDK VM root filesystem contains core packages that are configured to optimize running the CDK VM and containers. The available storage on the root filesystem is determined by the overlay size, which is smaller than the total available storage.
Workaround: Avoid installing packages or storing large files in the root filesystem of the CDK VM. Instead, you can create a sub-directory in the /mnt/sda1/ persistent storage volume or define and mount host folders that can share storage space between the host and the CDK VM.
If you want to perform development tasks inside the CDK VM, it is recommended that you use containers, which are stored in persistent storage volumes, and reuse the CDK Docker daemon.
4.3.3. Special characters cause passwords to fail
Depending on your operating system and shell environment, certain special characters can trigger variable interpolation and therefore cause passwords to fail.
Workaround: When creating and entering passwords, wrap the string with single quotes in the following format:
4.3.4. Cannot access web console with Microsoft Edge
Attempting to access the OpenShift web console using Microsoft Edge can result in the following error:
Site not reachable.
Alternatively, the OpenShift web console may be rendered as a blank page.
Cause: This is an inconsistent error caused by Microsoft Edge.
4.3.5. X.509 certificate is valid for 10.0.2.15, 127.0.0.1, 172.17.0.1, 172.30.0.1, 192.168.99.100, not 192.168.99.101
Starting a stopped CDK VM can produce the following X.509 certificate error:
$ minishift start -- Checking if requested hypervisor 'kvm' is supported on this platform ... OK -- Checking the ISO URL ... OK -- Starting local OpenShift cluster using 'kvm' hypervisor ... -- Starting Minishift VM ......................... OK [...] FAIL Error: cannot access master readiness URL https://192.168.99.101:8443/healthz/ready Details: No log available from "origin" container Caused By: Error: Get https://192.168.99.101:8443/healthz/ready: x509: certificate is valid for 10.0.2.15, 127.0.0.1, 172.17.0.1, 172.30.0.1, 192.168.99.100, not 192.168.99.101 Error during 'cluster up' execution: Error starting the cluster.
The reason for the above error is that OpenShift cluster certificates contain the IP of the CDK VM. The certificates are generated only when the CDK VM is freshly started. After restart, the CDK VM might be assigned a new IP address. If this happens, the certificate becomes invalid.
Workaround: Delete the existing CDK VM and start again.
$ minishift delete --force $ minishift start
4.3.6. Removing the subscription password from an OS-native keychain
On Windows, the password is stored using Credential Manager. Run the following command in Command Prompt to delete the stored password:
C:\> cmdkey /delete:minishift:<username>
126.96.36.199. Red Hat Enterprise Linux
On Red Hat Enterprise Linux, the password is stored using the D-Bus Secret Service API provided by libsecret. Run the following command to delete the stored password:
$ secret-tool clear service minishift username <username>
On macOS, the password is stored using Keychain Access. Run the following command to delete the stored password:
$ security delete-generic-password -s minishift