Refer to the following documents for further information:

To report issues, refer to the following:

Container Development Kit version 3.1 is based on the Minishift project version 1.5.0. Using a single executable file, the minishift command deploys Container Development Kit as a Red Hat Enterprise Linux virtual machine, running OpenShift (which includes Kubernetes, the docker service, and other container development and deployment software).

Container Development Kit setup procedure can, and should, be run as a regular user that has special permission to launch virtual machines. In the procedure, you will see how to assign that permission, along with ways to configure your hypervisor and command shell to start and effectively interact with Container Development Kit.

The following section describes how to install Container Development Kit and the required dependencies.

Container Development Kit requires a hypervisor to start the virtual machine on which the OpenShift cluster is provisioned. Verify that the hypervisor of your choice is installed and enabled on your system before you set up Container Development Kit. Once the hypervisor is up and running, additional setup is required for Container Development Kit to work with that hypervisor (as described in the coming setup procedure).

Depending on your host operating system, you have the choice of the following hypervisors:

Refer to the documentation for each hypervisor to determine the hardware and operating system versions needed to run that hypervisor.

These are the basic steps for setting up Container Development Kit on your personal laptop or desktop system:

The following steps describe how to prepare your virtualization environment (hypervisor) for CDK, download CDK software, set up CDK, and start using it.

Follow the appropriate procedure to set up virtualization for your particular operating system and hypervisor. Container Development Kit uses Docker Machine and its driver plug-in architecture to provide a consistent way to manage the OpenShift VM.

Some hypervisors require that the driver plug-in be manually installed. Container Development Kit embeds VirtualBox drivers so no additional steps are required to configure the driver. However, later you will need to run a minishift command to tell Container Development Kit to use VirtualBox.

$ go get -u -d github.com/zchee/docker-machine-driver-xhyve
$ cd $GOPATH/src/github.com/zchee/docker-machine-driver-xhyve

# Install docker-machine-driver-xhyve binary into /usr/local/bin
$ make install

# docker-machine-driver-xhyve need root owner and uid
$ sudo chown root:wheel /usr/local/bin/docker-machine-driver-xhyve
$ sudo chmod u+s /usr/local/bin/docker-machine-driver-xhyve

Before you can download CDK software, you need to either register with the Red Hat Developer Program site or login to the Red Hat customer portal with Red Hat subscription credentials. Then go to one of the following two sites and download the software associated with your operating system:

Copy the downloaded minishift file to a directory in your $PATH and make it executable. The downloaded minishift executable is named cdk-3.1.0-1-minishift-darwin-amd64 (for macOS), cdk-3.1.0-1-minishift-linux-amd64 (for Linux) or cdk-3.1.0-1-minishift-windows-amd64.exe (for Windows). Assuming minishift executable is in the Downloads directory, follow the procedure for your operating system:

For Red Hat Enterprise Linux:

$ mkdir -p ~/bin
$ cp ~/Downloads/cdk-3.1.0-1-minishift* ~/bin/minishift
$ chmod +x ~/bin/minishift
$ export PATH=$PATH:$HOME/bin
$ echo ‘export PATH=$PATH:$HOME/bin’ >> ~/.bashrc

For macOS:

$ mkdir -p ~/bin
$ cp ~/Downloads/cdk-3.1.0-1-minishift* ~/bin/minishift
$ chmod +x ~/bin/minishift
$ export PATH=$PATH:$HOME/bin
$ echo export PATH=$PATH:$HOME/bin >> ~/.bash_profile

For Windows:

Create the desired directory and copy the downloaded minishift binary to the directory, renaming the binary to minishift. Add the directory path to the Windows path variable. This directory MUST be on the C: drive!

If it’s difficult to get minishift in your PATH, you can simply run it from the current directory as ./minishift (or .\minishift in some Windows shells).

The minishift setup-cdk command gets and configures the components needed to run Container Development Kit on your system. By default, it places Container Development Kit content in your ~/.minishift directory. One thing you should know about this directory:

Run the following command to set up Container Development Kit for Red Hat Enterprise Linux:

$ minishift setup-cdk
Setting up CDK 3 on host using '/home/joe/.minishift' as Minishift's home directory
Copying minishift-rhel7.iso to '/home/joe/.minishift/cache/iso/minishift-rhel7.iso'
Copying oc to '/home/joe/.minishift/cache/oc/v3.6.173.0.21/oc'
Creating configuration file '/home/joe/.minishift/config/config.json'
Creating marker file '/home/joe/.minishift/cdk'
Default add-ons anyuid, admin-user, xpaas installed
Default add-ons anyuid, admin-user, xpaas enabled
CDK 3 setup complete.

As you can see from the output, a .minishift directory is created in the user’s home directory to hold various CDK components, and appropriate files and setting are stored there.

For Windows or macOS: Running the same minishift setup-cdk command on Windows and Mac results in slightly different output, based on some different components and pathnames.

The minishift start command launches Container Development Kit, which consists of a Red Hat Enterprise Linux virtual machine running OpenShift. Follow these steps to launch Container Development Kit.

For macOS and Windows system, you see similar output from minishift start when run in a macOS terminal or Windows command prompt or PowerShell, respectively.

With CDK virtual machine running, you can configure and start using CDK through two primary interfaces: oc (OpenShift client binary command) or the OpenShift web console.

For more information about interacting with OpenShift from the command-line interface, see the OpenShift Client Binary section.

You are now ready to start using OpenShift using either of those two interfaces.

OpenShift provides various sample applications, such as templates, builder applications, and quickstarts. To begin creating an application, see :

The following steps describe how to deploy a sample Node.js application from the command-line.

See minishift stop for details on managing subscriptions as you stop and start the VM.

For more information about creating applications in OpenShift, see Creating New Applications in the OpenShift documentation.

This section describes how you can uninstall the minishift binary and delete associated files.

The minishift start command creates an OpenShift cluster using the cluster up approach. For this purpose it copies the oc binary onto your host.

The oc binary is located in the ~/.minishift/cache/oc/v1.5.1 directory, assuming that you use CDK’s default version of OpenShift. You can add this binary to your PATH using minishift oc-env, which displays the command you need to type into your shell.

The output of minishift oc-env differs depending on the operating system and the shell type.

$ minishift oc-env
export PATH="/Users/john/.minishift/cache/oc/v3.6.173.0.21:$PATH"
# Run this command to configure your shell:
# eval $(minishift oc-env)

As a part of the minishift start command, a Container Development Kit CLI profile is also created. This profile, also known as a context, contains the configuration to communicate with your OpenShift cluster.

Container Development Kit activates this context automatically, but if you need to switch back to it after, for example, logging into another OpenShift instance, you can run:

$ oc config use-context minishift

For an introduction to oc usage, see the Get Started with the CLI topic in the OpenShift documentation.

By default, cluster up uses AllowAllPasswordIdentityProvider to authenticate against the local cluster. This means any non-empty user name and password can be used to login to the local cluster.

The recommended user name and password is developer and developer, respectively. This is because they are already assigned to the default project myproject and also can impersonate the administrator. This allows you to run administrator commands using the --as system:admin parameter.

To login as administrator, use the system account:

$ oc login -u system:admin

In this case, client certificates are used. The certificates are stored in ~/.kube/config. The cluster up command installs the appropriate certificates as a part of the bootstrap.

To view the available login contexts, run:

$ oc config view

To access the OpenShift Web console, you can run this command in a shell after running the minishift command to get the URL of the Web console:

$ minishift console --url

Alternatively, after starting Container Development Kit, you can use the following command to directly open the console in a browser:

$ minishift console

To access a service that is exposed with a route, run this command in a shell:

$ minishift openshift service [-n NAMESPACE] [--url] NAME

To list all the services in the OpenShift instance, run this command in a shell:

$ minishift openshift service list

For more information on exposing services refer also to Exposing Services.

To access OpenShift logs, run the logs command after starting Container Development Kit:

$ minishift logs

While OpenShift is running, you can view and change the master or the node configuration of your cluster.

To view the OpenShift master configuration file master-config.yaml, run the following command:

$ minishift openshift config view

To show the node configuration instead of the master configuration, specify the target flag.

$  minishift openshift config set --patch '{"corsAllowedOrigins": [".*"]}'

$ minishift openshift config set --patch '{\"corsAllowedOrigins\": [\".*\"]}'

$ minishift openshift config set --patch "{\"corsAllowedOrigins\": [\".*\"]}"

$ minishift openshift config set --patch '{"routingConfig": {"subdomain": "<IP-ADDRESS>.xip.io"}}'

There are several ways you can expose your service after you deploy it on OpenShift. The following sections describes the various methods and when to use them.

If you are deploying a Web application, the most common way to expose it is by a route. A route exposes the service as a host name. You can create a route using the Web console or the CLI:

oc expose svc/frontend --hostname=www.example.com

To see a full example of creating an application and exposing it with a route, see the Deploying a Sample Application section.

In case the service you want to expose is not HTTP based, you can create a NodePort service. In this case, each OpenShift node will proxy that port into your service. To access this port on your Container Development Kit VM, you need to configure an Ingress IP Self-Service using oc expose with the parameter type=LoadBalancer.

A common use-case for Ingress IP Self-Service is the ability to expose a database service. The following example shows the complete workflow to create and expose a MariaDB instance using the minishift and oc commands:

$ minishift start
$ eval $(minishift oc-env)
$ oc new-app -e MYSQL_ROOT_PASSWORD=admin https://raw.githubusercontent.com/openshift/origin/master/examples/db-templates/mariadb-persistent-template.json
$ oc rollout status -w dc/mariadb
$ oc expose dc mariadb --type=LoadBalancer --name=mariadb-ingress
$ oc export svc mariadb-ingress
 ....
ports:
    - nodePort: 30907
 ....

After the service is exposed, you can access MariaDB with the mysql CLI using Container Development Kit VM IP and the exposed NodePort service.

$ mysql --user=root --password=foo --host=$(minishift ip) --port=30907

If you want to quickly access a port of a specific pod of your cluster, you can also use the oc port-forward command of the OpenShift CLI.

$ oc port-forward POD [LOCAL_PORT:]REMOTE_PORT

OpenShift provides an integrated Docker registry which can be used for development as well. Images present in the registry can directly be used for applications, speeding up the local development workflow.

The following example shows how to deploy an OpenShift application directly from a locally-built docker image. This example uses the OpenShift project myproject. This project is automatically created by minishift start.

When you use the minishift command, you interact with two components:

The following sections contain information about managing Container Development Kit VM. For details about using Container Development Kit to manage your local OpenShift cluster, see the Interacting with OpenShift section.

Use different options to the minishift command to manage Container Development Kit throughout its life-cycle.

The runtime behavior of the minishift command can be controlled through flags, environment variables, and persistent configuration options.

The following precedence order is applied to control the behavior of the minishift command. Each action in the following list takes precedence over the action below it:

$ minishift setup-cdk --minishift-home ~/.mynewdir
$ export MINISHIFT_HOME=~/.mynewdir
$ echo "export MINISHIFT_HOME=~/.mynewdir" >> ~/.bashrc

# Set default memory 4096 MB
$ minishift config set memory 4096

$ minishift start --insecure-registry hub.foo.com --insecure-registry hub.bar.com

$ minishift config set insecure-registry hub.foo.com,hub.bar.com

$ minishift start --vm-driver virtualbox

$ minishift config set vm-driver virtualbox

$ minishift config view
- memory: 4096

$ minishift config get memory
4096

$ minishift config unset memory

$ export XHYVE_EXPERIMENTAL_NFS_SHARE=true
$ minishift start --vm-driver xhyve

To speed up provisioning of the OpenShift cluster and to minimize network traffic, the core OpenShift images can be cached on the host. This feature is considered experimental and needs to be explicitly enabled using the minishift config set command:

$ minishift config set image-caching true

Once enabled, caching occurs transparently, in a background process, the first time you use the minishift start command. Once the images are cached under $MINISHIFT_HOME/cache/images, successive Container Development Kit VM creations will use these cached images.

Each time an image exporting background process runs, a log file is generated under $MINISHIFT_HOME/logs which can be used to verify the progress of the export.

You can disable the caching of the OpenShift images by setting image-caching to false or removing the setting altogether using minishift config unset:

$ minishift config unset image-caching

As part of the OpenShift cluster provisioning, 100 persistent volumes are created for your OpenShift cluster. This allows applications to make persistent volumes claims. The location of the persistent data is determined in the host-pv-dir flag of the minishift start command and defaults to /var/lib/minishift/openshift.local.pv on Container Development Kit VM.

If you are behind an HTTP/HTTPS proxy, you need to supply proxy options to allow Docker and OpenShift to work properly. To do this, pass the required flags during minishift start.

For example:

$ minishift start --http-proxy http://YOURPROXY:PORT --https-proxy https://YOURPROXY:PORT

In an authenticated proxy environment, the proxy_user and proxy_password must be a part of proxy URI.

 $ minishift start --http-proxy http://<proxy_username>:<proxy_password>@YOURPROXY:PORT \
                   --https-proxy https://<proxy_username>:<proxy_password>@YOURPROXY:PORT

You can also use the --no-proxy flag to specify a comma-separated list of hosts that should not be proxied.

Using the proxy options will transparently configure the docker daemon as well as OpenShift to use the specified proxies.

The Container Development Kit VM is exposed to the host system with a host-only IP address that can be obtained with the minishift ip command.

You can use the minishift ssh command to interact with Container Development Kit VM.

You can run minishift ssh without a sub-command to open an interactive shell and run commands on Container Development Kit VM in the same way that you run commands interactively on any remote machine using SSH.

You can also run minishift ssh with a sub-command to send the sub-command directly to Container Development Kit VM and return the result to your local shell. For example:

$ minishift ssh -- docker ps
CONTAINER ID  IMAGE                     COMMAND        CREATED        STATUS        NAMES
71fe8ff16548  .../ose-pod:v3.6.173.0.21 "/usr/bin/pod" 4 minutes ago  Up 4 minutes  k8s_POD...

Container Development Kit allows you to extend the vanilla OpenShift setup provided by cluster up with an add-on mechanism.

Add-ons are directories that contain a text file with the .addon extension. The directory can also contain other resource files such as JSON template files. However, only one .addon file is allowed per add-on.

The following example shows the contents of an add-on file, including the name and description of the add-on, additional metadata, and the actual add-on commands to apply.

# Name: anyuid                                                                          1
# Description: Allows authenticated users to run images under a non pre-allocated UID   2
# Required-vars: ACME_TOKEN                                                             3

oc adm policy add-scc-to-group anyuid system:authenticated                              4

Enabled add-ons are applied during minishift start, immediately after the initial cluster provisioning is successfully completed.

This section describes the commands that an add-on file can contain. They are forming a mini-DSL for Container Development Kit add-ons:

Container Development Kit allows the use of variables within the add-on commands. Variables have the format #{<variable-name>}. The following example shows how the OpenShift routing suffix can be interpolated into an openshift command to create a new certificate as part of securing the OpenShift registry. The used variable #{routing-suffix} is part of the built-in add-on variables.

$ openshift admin ca create-server-cert \
  --signer-cert=/var/lib/origin/openshift.local.config/master/ca.crt \
  --signer-key=/var/lib/origin/openshift.local.config/master/ca.key \
  --signer-serial=/var/lib/origin/openshift.local.config/master/ca.serial.txt \
  --hostnames='docker-registry-default.#{routing-suffix},docker-registry.default.svc.cluster.local,172.30.1.1' \
  --cert=/etc/secrets/registry.crt \
  --key=/etc/secrets/registry.key

$ minishift addons apply --addon-env PROJECT_USER=john acme

$ minishift config set addon-env PROJECT_USER=john
$ minishift addons apply acme

$ minishift config set addon-env PROJECT_USER=env.USER        1
$ minishift addons apply acme                                 2

# Name: acme
# Description: ACME add-on
# Required-Vars: PROJECT_USER

Container Development Kit provides a set of built-in add-ons that offer some common OpenShift customization to assist with development. During minishift setup-cdk, minishift automatically installs and enables xpaas, anyuid, and admin-user add-ons. To install the default add-ons, run:

$ minishift addons install --defaults

This command extracts the default add-ons to the add-on installation directory $MINISHIFT_HOME/addons. To view the list of installed add-ons, you can then run:

$ minishift addons list --verbose=true

This command prints a list of installed add-ons. You should at least see the anyuid add-on listed. This is an important add-on that allows you to run images that do not use a pre-allocated UID. By default, this is not allowed in OpenShift.

Add-ons are enabled with the minishift addons enable command and disabled with the minishift addons disable command. Enabled add-ons automatically get executed during minishift start.

The following examples show how to enable and disable the anyuid add-on.

$ minishift addons enable anyuid

$ minishift addons disable anyuid

$ minishift addons enable registry --priority=5

$ minishift addons list
- anyuid         : enabled    P(0)
- registry       : enabled    P(5)
- eap            : enabled    P(10)

Add-ons can be explicitly executed with the minishift addons apply command. You can use the apply command for both enabled and disabled add-ons. To apply multiple add-ons with a single command, specify add-on names separated by space.

The following example shows how to explicitly apply the anyuid and the admin-user add-ons.

$ minishift addons apply anyuid admin-user

To write a custom add-on, you should create a directory and in it create one (and only one) text file with the extension .addon, for example admin-role.addon.

This file needs to contain the Name and Description metadata fields, as well as the commands that you want to execute as a part of the add-on.

The following example shows the definition of an add-on that gives the developer user cluster-admin privileges.

# Name: admin-role
# Description: Gives the developer user cluster-admin privileges

oc adm policy add-role-to-user cluster-admin developer

After you define the add-on, you can install it by running:

$ minishift addons install <ADDON_DIR_PATH>

Host folders are directories on the host which are shared between the host and Container Development Kit VM. They allow for a two way file synchronization between host and VM. The following sections discuss the various types of host folders, driver provided host folders, as well as the minishift hostfolder command.

Some drivers mount a default host folder into the VM in order to share files between the VM and the host. These folders are currently not configurable and differ for each driver and OS.

Container Development Kit provides the minishift hostfolder command to list, add, mount, unmount and remove host folders. In contrast to the driver-provided host folders, you can use the hostfolder command to mount multiple shared folders onto custom specified mount points.

$ minishift hostfolder list
Name        Mountpoint            Remote path              Mounted
myshare     /mnt/sda1/myshare     //192.168.1.82/MYSHARE   N

$ minishift hostfolder add myshare 1
UNC path: //192.168.99.1/MYSHARE   2
Mountpoint [/mnt/sda1/myshare]:    3
Username: john                     4
Password: [HIDDEN]                 5
Domain:                            6
Added: myshare

$ minishift hostfolder mount myshare
Mounting 'myshare': '//192.168.99.1/MYSHARE' as '/mnt/sda1/myshare' ... OK

$ minishift hostfolder list
Name       Mountpoint          Remote path              Mounted
myshare    /mnt/sda1/myshare   //192.168.99.1/MYSHARE   Y

$ minishift ssh "ls -al /mnt/sda1/myshare"

$ minishift config set hostfolders-automount true

$ minishift hostfolder umount myshare
Unmounting 'myshare' ... OK

$ minishift hostfolder list
Name       Mountpoint          Remote path              Mounted
myshare    /mnt/sda1/myshare   //192.168.99.1/MYSHARE   N

$ minishift hostfolder list
Name        Mountpoint            Remote path              Mounted
myshare     /mnt/sda1/myshare     //192.168.1.82/MYSHARE   N

$ minishift hostfolder remove myshare
Removed: myshare

$ minishift hostfolder list
No host folders defined

When running OpenShift in a single VM, you can reuse the docker daemon managed by Container Development Kit for other docker use-cases as well. By using the same docker daemon as the one used in Container Development Kit, you can speed up your local development.

In order to configure your console to reuse Container Development Kit docker daemon, follow these steps:

This section contains solutions to common problems that you might encounter while using Container Development Kit.

Depending on your operating system and shell environment, certain special characters can trigger variable interpolation and therefore cause passwords to fail.

Workaround: When creating and entering passwords, wrap the string with single quotes in the following format: '<password>'

If you use virsh on KVM/libvirt to create snapshots in your development workflow, and then use minishift delete to delete the snapshots along with the VM, you might encounter the following error:

$ minishift delete
Deleting the Minishift VM...
Error deleting the VM:  [Code-55] [Domain-10] Requested operation is not valid: cannot delete inactive domain with 4 snapshots

Cause: The snapshots are stored in ~/.minishift/machines, but the definitions are stored in /var/lib/libvirt/qemu/snapshot/minishift.

Workaround: To delete the snapshots, you need to perform the following steps as root.

$ rm -rf ~/.minishift/machines

It is recommended to avoid using metadata when you create snapshots. To ensure this, you can specify the --no-metadata flag. For example:

# virsh snapshot-create-as --domain vm1 overlay1 --diskspec vda,file=/export/overlay1.qcow2 --disk-only --atomic --no-metadata

The problem is likely that the libvirtd service is not running. You can check this with the following command:

$ systemctl status libvirtd

If libvirtd is not running, start it and enable it to start on boot:

$ systemctl start libvirtd
$ systemctl enable libvirtd

The problem is likely that the virtlogd service is not running. You can check this with the following command:

$ systemctl status virtlogd

If virtlogd is not running, start it and enable it to start on boot:

$ systemctl start virtlogd
$ systemctl enable virtlogd

If you try minishift start and the this error appears, ensure that you use minishift delete to delete the VMs that you created earlier. However, if this fails and you want to completely clean up Container Development Kit and start fresh, do the following:

In case all of this fails, you might want to uninstall Container Development Kit and do a fresh install of Container Development Kit.

The problem is likely that the xhyve driver is not able to clean up vmnet when a VM is removed. vmnet.framework determines the IP address based on the following files:

Reset Container Development Kit-specific IP database, ensure that you remove the minishift entry section from the dhcpd_leases file, and reboot your system.

{
  ip_address=192.168.64.2
  hw_address=1,2:51:8:22:87:a6
  identifier=1,2:51:8:22:87:a6
  lease=0x585e6e70
  name=minishift
}

If you use Windows, ensure that you set the --vm-driver virtualbox flag in the minishift start command. Alternatively, the problem might be an outdated version of VirtualBox.

To avoid this issue, it is recommended to use VirtualBox 5.1.12 or later.

If you run Container Development Kit with Hyper-V on Windows as a normal user or as a user with Administrator privileges, you might encounter the following error:

Error starting the VM: Error creating the VM. Error with pre-create check: "Hyper-V commands must be run as an Administrator".

Workaround: You can either add yourself to the Hyper-V Administrators group, which is recommended, or run the shell in an elevated mode.

If you are using PowerShell, you can add yourself to the Hyper-V Administrators group as follows:

You can also use the GUI to add yourself to the Hyper-V Administrators group as follows:

Now you can run the Hyper-V commands as a normal user.

For more options for Hyper-V see creating Hyper-V administrators local group.

If you try to use Container Development Kit with Hyper-V using an external virtual switch while you are connected to a VPN such as OpenVPN, Container Development Kit might fail to provision the VM.

Cause: Hyper-V networking might not route the network traffic in both directions properly when connected to a VPN.

Workaround: Disconnect from the VPN and try again after stopping the VM from the Hyper-V manager.