Introduction

Kubeflow makes life simpler by allowing deployment of Machine Learning workflows on Kubernetes. This brings in the flexibility of scale and portability.

In this tutorial you will learn: 

1. Provisioning of a Kubernetes cluster in IBM Cloud and installing the required tools
2. Installing Kubeflow
3. Launching the Kubeflow dashboard

Prerequisites

To run this tutorial, you need:

1. An Ubuntu 18 machine with a minimum 8 cores, 16 GB RAM, and 250 GB storage
2. Root privilege on the ubuntu machine
3. IBM Cloud account

Deployment Setup

IBM Cloud UI

Log in to IBM Cloud ( https://cloud.ibm.com ). Go to Catalog, under services you can find Kubernetes service.

In this demo, I have used the Kubernetes 1.16.14 (Stable) version in a classic infrastructure in a single zone region. Once you choose the version, select Geography ( Asia Pacific), Availability (Single Zone), Worker Zone (Chennai 01) 

The worker node size used in this setup was 4 vCPUs and 16  GB Memory with 1 worker node per zone.

The master service endpoint is set to Public endpoint only.

Choose the cluster name wisely, you can leave it default but recommend to use something of your choice.

Create the cluster 

Ubuntu Machine

Till the cluster setup is up, log in to the Ubuntu Machine 

Download the necessary tools for IBM Cloud.

Shell1

curl -sL https://ibm.biz/idt-installer | bash

Login in to your IBM Cloud Account, the region, and resource group might change based on your selection.

Shell

1

ibmcloud login -a cloud.ibm.com -r jp-tok -g default

Set the Kubernetes context to your cluster for this terminal session.

Shell

1

ibmcloud ks cluster config --cluster "YOUR CLUSTER ID"

Verify that you can connect to your cluster.

Shell

1

kubectl config current-context

Note: In case you forget your cluster setting, you can retrieve this information by visiting https://cloud.ibm.com/kubernetes/clusters and clicking Access under your cluster.

To verify your setting, run the following code.

Shell

1

kubectl version --short

The following should be your output.

Shell

1

Client Version: v1.17.11

2

Server Version: v1.16.14+IKS

IBM Cloud Block Storage Setup

When using the classic worker nodes provider of IBM Cloud Kubernetes cluster, by default, it uses IBM Cloud File Storage based on NFS as the default storage class. File Storage is designed to run RWX (read-write multiple nodes) workloads with proper security built around it. Therefore, File Storage does not allow group securityContext which is needed for DEX and Kubeflow Jupyter Server.

IBM Cloud Block Storage provides a fast way to store data and satisfy many of the Kubeflow persistent volume requirements such as fsGroup out of the box and optimized RWO (read-write single node) which is used on all Kubeflow’s persistent volume claim.

Check if there is a default storage class setup to Block Storage.

Shell

1

kubectl get storageclass | grep \(default\)

If not, we need to set up IBM Cloud storage.

Follow the instructions to install the Helm version 3 client on your local machine.

Add the IBM Cloud Helm chart repository to the cluster where you want to use the IBM Cloud Block Storage plug-in.

Shell

1

helm repo add iks-charts https://icr.io/helm/iks-charts

2

helm repo update

3

helm install 1.6.0 iks-charts/ibmcloud-block-storage-plugin -n kube-system

The following code shows the sample output.

Shell

1

NAME: 1.6.0

2

LAST DEPLOYED: Fri Mar 20 11:38:21 2020

3

NAMESPACE: kube-system

4

STATUS: deployed

5

REVISION: 1

6

NOTES:

7

Thank you for installing: ibmcloud-block-storage-plugin.   Your release is named: 1.6.0

Set the block storage as default, ensure that there is only one default storage

Shell

1

kubectl patch storageclass ibmc-block-gold -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

2

3

# Check the default storageclass is block storage

4

kubectl get storageclass | grep \(default\)

Installing Kubeflow v.1.1.0

You can create your directory, I have created a kubeflow directory under root. 

Shell

1

mkdir -p /root/kubeflow/v1.1.0

2

cd /root/kubeflow/v1.1.0

3

wget https://github.com/kubeflow/kfctl/releases/download/v1.1.0/kfctl_v1.1.0-0-g9a3621e_linux.tar.gz

4

tar -xvf kfctl_v1.0.1-0-gf3edb9b_linux.tar.gz

5

export PATH=$PATH:/root/kubeflow/v1.1.0

6

export KF_NAME=my-kubeflow

7

export BASE_DIR=/root/kubeflow/v1.1.0

8

export KF_DIR=${BASE_DIR}/${KF_NAME}

9

# config file is IBM specific

10

export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_ibm.v1.1.0.yaml"

11

12

mkdir -p ${KF_DIR}

13

cd ${KF_DIR}

14

kfctl apply -V -f ${CONFIG_URI}

When the installation is complete which might take a few mins, please ensure that all pods are running. Only then proceed to the next steps. If not you might have to recreate this exercise.

Verify if all pods are running:

Shell

1

kubectl get pod -n kubeflow

The sample output below:

Launching the Kubeflow Dashboard

Kubeflow Dashboard can be accessed through the istio-ingressgateway service. To see your setting of istio-ingressgateway service, execute the following code.

Shell

1

kubectl get -n istio-system svc/istio-ingressgateway

You might get the following output:

Shell

1

NAME                   TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)                                                                                                                                     AGE

2

istio-ingressgateway   NodePort   111.11.11.111   <none>       TCP

If your output has a NodePort TYPE (not LoadBalancer) and EXTERNAL-IP is blank like above, run the following command to change the type from NodePort to LoadBalancer. 

Shell

1

kubectl edit -n istio-system svc/istio-ingressgateway

Run the following command again to get the EXTERNAL-IP.

Shell

1

kubectl get -n istio-system svc/istio-ingressgateway

Now you can access the Kubeflow dashboard in a web browser: http://<EXTERNAL-IP>:<PORT>

Summary

That's all folks! In this tutorial, I have explained to you how to set up Kubeflow on IBM Kubernetes Service (IKS). I hope you liked it.