Chapter 3. Using Container Development Kit

When you use the minishift command, you interact with two components:

The following sections contain information about managing Container Development Kit VM. For details about using Container Development Kit to manage your local OpenShift cluster, see the Interacting with OpenShift section.

Use different options to the minishift command to manage Container Development Kit throughout its life-cycle.

The runtime behavior of the minishift command can be controlled through flags, environment variables, and persistent configuration options.

The following precedence order is applied to control the behavior of the minishift command. Each action in the following list takes precedence over the action below it:

$ minishift setup-cdk --minishift-home ~/.mynewdir
$ export MINISHIFT_HOME=~/.mynewdir
$ echo "export MINISHIFT_HOME=~/.mynewdir" >> ~/.bashrc

# Set default memory 4096 MB
$ minishift config set memory 4096

$ minishift start --insecure-registry hub.foo.com --insecure-registry hub.bar.com

$ minishift config set insecure-registry hub.foo.com,hub.bar.com

$ minishift start --vm-driver virtualbox

$ minishift config set vm-driver virtualbox

$ minishift config view
- memory: 4096

$ minishift config get memory
4096

$ minishift config unset memory

$ export XHYVE_EXPERIMENTAL_NFS_SHARE=true
$ minishift start --vm-driver xhyve

To speed up provisioning of the OpenShift cluster and to minimize network traffic, the core OpenShift images can be cached on the host. This feature is considered experimental and needs to be explicitly enabled using the minishift config set command:

$ minishift config set image-caching true

Once enabled, caching occurs transparently, in a background process, the first time you use the minishift start command. Once the images are cached under $MINISHIFT_HOME/cache/images, successive Container Development Kit VM creations will use these cached images.

Each time an image exporting background process runs, a log file is generated under $MINISHIFT_HOME/logs which can be used to verify the progress of the export.

You can disable the caching of the OpenShift images by setting image-caching to false or removing the setting altogether using minishift config unset:

$ minishift config unset image-caching

As part of the OpenShift cluster provisioning, 100 persistent volumes are created for your OpenShift cluster. This allows applications to make persistent volumes claims. The location of the persistent data is determined in the host-pv-dir flag of the minishift start command and defaults to /var/lib/minishift/openshift.local.pv on Container Development Kit VM.

If you are behind an HTTP/HTTPS proxy, you need to supply proxy options to allow Docker and OpenShift to work properly. To do this, pass the required flags during minishift start.

For example:

$ minishift start --http-proxy http://YOURPROXY:PORT --https-proxy https://YOURPROXY:PORT

In an authenticated proxy environment, the proxy_user and proxy_password must be a part of proxy URI.

 $ minishift start --http-proxy http://<proxy_username>:<proxy_password>@YOURPROXY:PORT \
                   --https-proxy https://<proxy_username>:<proxy_password>@YOURPROXY:PORT

You can also use the --no-proxy flag to specify a comma-separated list of hosts that should not be proxied.

Using the proxy options will transparently configure the docker daemon as well as OpenShift to use the specified proxies.

The Container Development Kit VM is exposed to the host system with a host-only IP address that can be obtained with the minishift ip command.

You can use the minishift ssh command to interact with Container Development Kit VM.

You can run minishift ssh without a sub-command to open an interactive shell and run commands on Container Development Kit VM in the same way that you run commands interactively on any remote machine using SSH.

You can also run minishift ssh with a sub-command to send the sub-command directly to Container Development Kit VM and return the result to your local shell. For example:

$ minishift ssh -- docker ps
CONTAINER ID  IMAGE                     COMMAND        CREATED        STATUS        NAMES
71fe8ff16548  .../ose-pod:v3.6.173.0.21 "/usr/bin/pod" 4 minutes ago  Up 4 minutes  k8s_POD...

Container Development Kit allows you to extend the vanilla OpenShift setup provided by cluster up with an add-on mechanism.

Add-ons are directories that contain a text file with the .addon extension. The directory can also contain other resource files such as JSON template files. However, only one .addon file is allowed per add-on.

The following example shows the contents of an add-on file, including the name and description of the add-on, additional metadata, and the actual add-on commands to apply.

# Name: anyuid                                                                          1
# Description: Allows authenticated users to run images under a non pre-allocated UID   2
# Required-vars: ACME_TOKEN                                                             3

oc adm policy add-scc-to-group anyuid system:authenticated                              4

Enabled add-ons are applied during minishift start, immediately after the initial cluster provisioning is successfully completed.

This section describes the commands that an add-on file can contain. They are forming a mini-DSL for Container Development Kit add-ons:

Container Development Kit allows the use of variables within the add-on commands. Variables have the format #{<variable-name>}. The following example shows how the OpenShift routing suffix can be interpolated into an openshift command to create a new certificate as part of securing the OpenShift registry. The used variable #{routing-suffix} is part of the built-in add-on variables.

$ openshift admin ca create-server-cert \
  --signer-cert=/var/lib/origin/openshift.local.config/master/ca.crt \
  --signer-key=/var/lib/origin/openshift.local.config/master/ca.key \
  --signer-serial=/var/lib/origin/openshift.local.config/master/ca.serial.txt \
  --hostnames='docker-registry-default.#{routing-suffix},docker-registry.default.svc.cluster.local,172.30.1.1' \
  --cert=/etc/secrets/registry.crt \
  --key=/etc/secrets/registry.key

$ minishift addons apply --addon-env PROJECT_USER=john acme

$ minishift config set addon-env PROJECT_USER=john
$ minishift addons apply acme

$ minishift config set addon-env PROJECT_USER=env.USER        1
$ minishift addons apply acme                                 2

# Name: acme
# Description: ACME add-on
# Required-Vars: PROJECT_USER

Container Development Kit provides a set of built-in add-ons that offer some common OpenShift customization to assist with development. During minishift setup-cdk, minishift automatically installs and enables xpaas, anyuid, and admin-user add-ons. To install the default add-ons, run:

$ minishift addons install --defaults

This command extracts the default add-ons to the add-on installation directory $MINISHIFT_HOME/addons. To view the list of installed add-ons, you can then run:

$ minishift addons list --verbose=true

This command prints a list of installed add-ons. You should at least see the anyuid add-on listed. This is an important add-on that allows you to run images that do not use a pre-allocated UID. By default, this is not allowed in OpenShift.

Add-ons are enabled with the minishift addons enable command and disabled with the minishift addons disable command. Enabled add-ons automatically get executed during minishift start.

The following examples show how to enable and disable the anyuid add-on.

$ minishift addons enable anyuid

$ minishift addons disable anyuid

$ minishift addons enable registry --priority=5

$ minishift addons list
- anyuid         : enabled    P(0)
- registry       : enabled    P(5)
- eap            : enabled    P(10)

Add-ons can be explicitly executed with the minishift addons apply command. You can use the apply command for both enabled and disabled add-ons. To apply multiple add-ons with a single command, specify add-on names separated by space.

The following example shows how to explicitly apply the anyuid and the admin-user add-ons.

$ minishift addons apply anyuid admin-user

To write a custom add-on, you should create a directory and in it create one (and only one) text file with the extension .addon, for example admin-role.addon.

This file needs to contain the Name and Description metadata fields, as well as the commands that you want to execute as a part of the add-on.

The following example shows the definition of an add-on that gives the developer user cluster-admin privileges.

# Name: admin-role
# Description: Gives the developer user cluster-admin privileges

oc adm policy add-role-to-user cluster-admin developer

After you define the add-on, you can install it by running:

$ minishift addons install <ADDON_DIR_PATH>

Host folders are directories on the host which are shared between the host and Container Development Kit VM. They allow for a two way file synchronization between host and VM. The following sections discuss the various types of host folders, driver provided host folders, as well as the minishift hostfolder command.

Some drivers mount a default host folder into the VM in order to share files between the VM and the host. These folders are currently not configurable and differ for each driver and OS.

Container Development Kit provides the minishift hostfolder command to list, add, mount, unmount and remove host folders. In contrast to the driver-provided host folders, you can use the hostfolder command to mount multiple shared folders onto custom specified mount points.

$ minishift hostfolder list
Name        Mountpoint            Remote path              Mounted
myshare     /mnt/sda1/myshare     //192.168.1.82/MYSHARE   N

$ minishift hostfolder add myshare 1
UNC path: //192.168.99.1/MYSHARE   2
Mountpoint [/mnt/sda1/myshare]:    3
Username: john                     4
Password: [HIDDEN]                 5
Domain:                            6
Added: myshare

$ minishift hostfolder mount myshare
Mounting 'myshare': '//192.168.99.1/MYSHARE' as '/mnt/sda1/myshare' ... OK

$ minishift hostfolder list
Name       Mountpoint          Remote path              Mounted
myshare    /mnt/sda1/myshare   //192.168.99.1/MYSHARE   Y

$ minishift ssh "ls -al /mnt/sda1/myshare"

$ minishift config set hostfolders-automount true

$ minishift hostfolder umount myshare
Unmounting 'myshare' ... OK

$ minishift hostfolder list
Name       Mountpoint          Remote path              Mounted
myshare    /mnt/sda1/myshare   //192.168.99.1/MYSHARE   N

$ minishift hostfolder list
Name        Mountpoint            Remote path              Mounted
myshare     /mnt/sda1/myshare     //192.168.1.82/MYSHARE   N

$ minishift hostfolder remove myshare
Removed: myshare

$ minishift hostfolder list
No host folders defined

When running OpenShift in a single VM, you can reuse the docker daemon managed by Container Development Kit for other docker use-cases as well. By using the same docker daemon as the one used in Container Development Kit, you can speed up your local development.

In order to configure your console to reuse Container Development Kit docker daemon, follow these steps:

This section contains solutions to common problems that you might encounter while using Container Development Kit.

Depending on your operating system and shell environment, certain special characters can trigger variable interpolation and therefore cause passwords to fail.

Workaround: When creating and entering passwords, wrap the string with single quotes in the following format: '<password>'

If you use virsh on KVM/libvirt to create snapshots in your development workflow, and then use minishift delete to delete the snapshots along with the VM, you might encounter the following error:

$ minishift delete
Deleting the Minishift VM...
Error deleting the VM:  [Code-55] [Domain-10] Requested operation is not valid: cannot delete inactive domain with 4 snapshots

Cause: The snapshots are stored in ~/.minishift/machines, but the definitions are stored in /var/lib/libvirt/qemu/snapshot/minishift.

Workaround: To delete the snapshots, you need to perform the following steps as root.

$ rm -rf ~/.minishift/machines

It is recommended to avoid using metadata when you create snapshots. To ensure this, you can specify the --no-metadata flag. For example:

# virsh snapshot-create-as --domain vm1 overlay1 --diskspec vda,file=/export/overlay1.qcow2 --disk-only --atomic --no-metadata

The problem is likely that the libvirtd service is not running. You can check this with the following command:

$ systemctl status libvirtd

If libvirtd is not running, start it and enable it to start on boot:

$ systemctl start libvirtd
$ systemctl enable libvirtd

The problem is likely that the virtlogd service is not running. You can check this with the following command:

$ systemctl status virtlogd

If virtlogd is not running, start it and enable it to start on boot:

$ systemctl start virtlogd
$ systemctl enable virtlogd

If you try minishift start and the this error appears, ensure that you use minishift delete to delete the VMs that you created earlier. However, if this fails and you want to completely clean up Container Development Kit and start fresh, do the following:

In case all of this fails, you might want to uninstall Container Development Kit and do a fresh install of Container Development Kit.

The problem is likely that the xhyve driver is not able to clean up vmnet when a VM is removed. vmnet.framework determines the IP address based on the following files:

Reset Container Development Kit-specific IP database, ensure that you remove the minishift entry section from the dhcpd_leases file, and reboot your system.

{
  ip_address=192.168.64.2
  hw_address=1,2:51:8:22:87:a6
  identifier=1,2:51:8:22:87:a6
  lease=0x585e6e70
  name=minishift
}

If you use Windows, ensure that you set the --vm-driver virtualbox flag in the minishift start command. Alternatively, the problem might be an outdated version of VirtualBox.

To avoid this issue, it is recommended to use VirtualBox 5.1.12 or later.

If you run Container Development Kit with Hyper-V on Windows as a normal user or as a user with Administrator privileges, you might encounter the following error:

Error starting the VM: Error creating the VM. Error with pre-create check: "Hyper-V commands must be run as an Administrator".

Workaround: You can either add yourself to the Hyper-V Administrators group, which is recommended, or run the shell in an elevated mode.

If you are using PowerShell, you can add yourself to the Hyper-V Administrators group as follows:

You can also use the GUI to add yourself to the Hyper-V Administrators group as follows:

Now you can run the Hyper-V commands as a normal user.

For more options for Hyper-V see creating Hyper-V administrators local group.

If you try to use Container Development Kit with Hyper-V using an external virtual switch while you are connected to a VPN such as OpenVPN, Container Development Kit might fail to provision the VM.

Cause: Hyper-V networking might not route the network traffic in both directions properly when connected to a VPN.

Workaround: Disconnect from the VPN and try again after stopping the VM from the Hyper-V manager.