Configure the Kubernetes cloud provider

• AWS
• GCP
•    Azure
• Kubernetes
• VMware Tanzu
• OpenShift
• On-premises

This document describes how to configure the Kubernetes provider for YugabyteDB universes using Yugabyte Platform. If no cloud providers are configured in the Yugabyte Platform console yet, the main Dashboard page prompts you to configure at least one cloud provider.

Prerequisites

To run YugabyteDB universes on Kubernetes, all you need to provide in the Yugabyte Platform console is your Kubernetes provider credentials. Yugabyte Platform uses those credentials to automatically provision and de-provision the pods that run YugabyteDB.

Before you install YugabyteDB on a Kubernetes cluster, perform the following:

• Create a yugabyte-platform-universe-management service account.
• Create a kubeconfig file of the earlier-created service account to configure access to the Kubernetes cluster.

Service account

This is the ServiceAccount whose secret can be used to generate a kubeconfig file. This account:

• Should not be deleted once it is in use by Yugabyte Platform.
• namespace in the ServiceAccount creation command can be replaced by the desired namespace in which to install YugabyteDB.

Run the following kubectl command to apply the YAML file:

Expect the following output:

serviceaccount/yugabyte-platform-universe-management created

You need to grant access to this ServiceAccount using ClusterRoles and Roles, as well as ClusterRoleBindings and RoleBindings, thus allowing it to manage the YugabyteDB universe's resources for you. Ensure that you have replaced the namespace from the commands with the correct namespace of the previously created ServiceAccount.

The tasks you can perform depend on your access level.

Global Admin can grant broad cluster level admin access by executing the following command:

Global Restricted can grant access to only the specific cluster roles to create and manage YugabyteDB universes across all the namespaces in a cluster using the following command:

This contains ClusterRoles and ClusterRoleBindings for the required set of permissions.

Validate the service account using the following command:

Namespace Admin can grant namespace-level admin access by using the following command:

If you have multiple target namespaces, then you have to apply the YAML in all of them.

Validate the service account using the following command:

Namespace Restricted can grant access to only the specific roles required to create and manage YugabyteDB universes in a particular namespace. Contains Roles and RoleBindings for the required set of permissions.

For example, if your goal is to allow the platform software to manage YugabyteDB universes in the namespaces yb-db-demo and yb-db-us-east4-a (the target namespaces), then you need to apply in both the target namespaces, as follows:

Validate the service account using the following command:

kubeconfig file for a Kubernetes cluster

You can create a kubeconfig file for previously created yugabyte-platform-universe-management service account as follows:

1. Run the following wget command to get the Python script for generating the kubeconfig file:

   wget https://raw.githubusercontent.com/YugaByte/charts/master/stable/yugabyte/generate_kubeconfig.py
   

2. Run the following command to generate the kubeconfig file:

   python generate_kubeconfig.py -s yugabyte-platform-universe-management -n <namespace>
   

   The following output should appear:

   Generated the kubeconfig file: /tmp/yugabyte-platform-universe-management.conf
   

3. Use this generated kubeconfig file as the kubeconfig in the Yugabyte Platform Kubernetes provider configuration.

Select the Kubernetes service

You can use the Pivotal Container Service or Managed Kubernetes Service.

Select the tab for the service you are using, as per the following illustration:

Use the configuration form shown in the following illustration to select the Kubernetes provider type from Type (Pivotal Container Service is the default):

Configure the cloud provider

Continue configuring your Kubernetes provider as follows:

• Give a meaningful name for your configuration.
• Choose one of the folloiwng ways to specify Kube Config for an availability zone:
  • Specify at provider level in the provider form. If specified, this configuration file is used for all availability zones in all regions.
  • Specify at zone level in the region form. This is required for multi-az or multi-region deployments.
• Use Service Account to provide the name of the service account which has necessary access to manage the cluster (see Create cluster).
• Use Image Registry to specify from where to pull YugabyteDB image. Accept the default setting, unless you are hosting the registry.
• Use the Pull Secret File field to upload the pull secret to download the image of the Enterprise YugabyteDB that is in a private repository. Your Yugabyte sales representative should have provided this secret.

The following illustration shows the completed form:

Configure region and zones

Continue configuring your Kubernetes provider by clicking Add Region and completing the Add new region dialog, as follows:

• Use the Region field to select the region.
• Use the Zone field to select a zone label that should match with your failure domain zone label failure-domain.beta.kubernetes.io/zone.
• Optionally, use the Storage Class field to enter a comma-delimited value. If you do not specify this value, it would default to standard. You need to ensure that this storage class exists in your Kubernetes cluster.
• Use the Namespace field to specify the namespace. If provided SA has the Cluster Admin permissions, you are not required to complete this field. The SA used in the provided kubeconfig file should have access to this namespace.
• Use Kube Config to upload the configuration file. If this file is available at provider level, you are not required to supply it.

• Complete the Overrides field using one of the provided options. If you do not specify anything, Yugabyte Platform would use defaults specified inside the Helm chart. The following overrides are available:

  • Overrides to add service-level annotations:

    serviceEndpoints:
      - name: "yb-master-service"
        type: "LoadBalancer"
        annotations:
          service.beta.kubernetes.io/aws-load-balancer-internal: "0.0.0.0/0"
        app: "yb-master"
        ports:
          ui: "7000"
    
      - name: "yb-tserver-service"
        type: "LoadBalancer"
        annotations:
          service.beta.kubernetes.io/aws-load-balancer-internal: "0.0.0.0/0"
        app: "yb-tserver"
        ports:
          ycql-port: "9042"
          yedis-port: "6379"
          ysql-port: "5433"
    

  • Overrides to disable LoadBalancer:

    enableLoadBalancer: False
    

  • Overrides to change the cluster domain name:

    domainName: my.cluster
    

  • Overrides to add annotations at StatefulSet-level:

    networkAnnotation:
      annotation1: 'foo'
      annotation2: 'bar'
    

  • Overrides to add custom resource allocation for YB master and TServer pods and it overrides the instance types selected in the Yugabyte universe creation flow:

    resource:
      master:
        requests:
          cpu: 2
          memory: 2Gi
        limits:
          cpu: 2
          memory: 2Gi
      tserver:
        requests:
          cpu: 2
          memory: 4Gi
        limits:
          cpu: 2
          memory: 4Gi
    

  • Overrides to enable Istio compatibility (required when Istio is used with Kubernetes):

    istioCompatibility: enabled: true
    

  • Overrides to publish Node-IP as the server broadcast address.

    By default, Master and T-Server pod fully-qualified domain names (FQDNs) are used within the cluster as the server broadcast address. To publish the IPs of the nodes on which YugabyteDB TServer pods are deployed, add the following YAML to each zone override configuration:

    tserver:
      extraEnv:
      - name: NODE_IP
        valueFrom:
          fieldRef:
            fieldPath: status.hostIP
      serverBroadcastAddress: "$(NODE_IP)"
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - "yb-tserver"
            topologyKey: kubernetes.io/hostname
    
    # Required to esure that the Kubernetes FQDNs are used for
    # internal communication between the nodes and node-to-node
    # TLS certificates are validated correctly.
    
    gflags:
      master:
        use_private_ip: cloud
      tserver:
        use_private_ip: cloud
    
    serviceEndpoints:
      - name: "yb-master-ui"
        type: LoadBalancer
        app: "yb-master"
        ports:
          http-ui: "7000"
    
      - name: "yb-tserver-service"
        type: NodePort
        externalTrafficPolicy: "Local"
        app: "yb-tserver"
        ports:
          tcp-yql-port: "9042"
          tcp-yedis-port: "6379"
          tcp-ysql-port: "5433"
    

Continue configuring your Kubernetes provider by clicking Add Zone and notice that there are might be multiple zones, as per the following illustration:

Finally, click Add Region, and then click Save to save the configuration. If successful, you will be redirected to the table view of all configurations.