The cluster-api-provider-incus book

LXC will spin up a plain Ubuntu instance with haproxy installed.

The controller will automatically update the list of backends on the haproxy configuration as control plane nodes are added or removed from the cluster.

No other configuration is required for “lxc” mode. The load balancer instance can be configured through the .instanceSpec field.

The load balancer container is a single point of failure to access the workload cluster control plane. Therefore, it should only be used for development or evaluation clusters.

OCI will spin up an OCI instance running the kindest/haproxy image.

The controller will automatically update the list of backends on the haproxy configuration as control plane nodes are added or removed from the cluster.

No other configuration is required for “oci” mode. The load balancer instance can be configured through the .instanceSpec field.

The load balancer container is a single point of failure to access the workload cluster control plane. Therefore, it should only be used for development or evaluation clusters.

Requires server extensions: instance_oci

OVN will create a network load balancer.

The controller will automatically update the list of backends for the network load balancer as control plane nodes are added or removed from the cluster.

The cluster administrator is responsible to ensure that the OVN network is configured properly and that the LXCMachineTemplate objects have appropriate profiles to use the OVN network.

When using the “ovn” mode, the load balancer address must be set in .spec.controlPlaneEndpoint.host on the LXCCluster object.

Requires server extensions: network_load_balancer, network_load_balancer_health_checks

KubeVIP will configure kube-vip on the control plane instances.

When using kube-vip, the controller will automatically inject /etc/kubernetes/manifests/kube-vip.yaml into all control plane nodes of the cluster.

When using the “kube-vip” mode, the load balancer address must be set in .spec.controlPlaneEndpoint.host on the LXCCluster object.

External will not create a load balancer. It must be used alongside something like kube-vip, otherwise the cluster will fail to provision.

When using the “external” mode, the load balancer address must be set in .spec.controlPlaneEndpoint.host on the LXCCluster object.

ControlPlaneEndpoint represents the endpoint to communicate with the control plane.

SecretRef references a secret with credentials to access the LXC (e.g. Incus, LXD) server.

LoadBalancer is configuration for provisioning the load balancer of the cluster.

Unprivileged will launch unprivileged LXC containers for the cluster machines.

Known limitations apply for unprivileged LXC containers (e.g. cannot use NFS volumes).

Do not apply the default kubeadm profile on container instances.

In this case, the cluster administrator is responsible to create the profile manually and set the .spec.template.spec.profiles field of all LXCMachineTemplate objects.

For more details on the default kubeadm profile that is applied, see https://capn.linuxcontainers.org/reference/profile/kubeadm.html

Ready denotes that the LXC cluster (infrastructure) is ready.

Conditions defines current service state of the LXCCluster.

V1Beta2 groups all status fields that will be added in LXCCluster’s status with the v1beta2 version.

Standard object’s metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

conditions represents the observations of a LXCCluster’s current state. Known condition types are Ready, LoadBalancerAvailable, Deleting, Paused.

InstanceSpec can be used to adjust the load balancer instance configuration.

CustomHAProxyConfigTemplate allows you to replace the default HAProxy config file content. Please use it with caution, as there are no checks to ensure the validity of the configuration.

Image is the kube-vip image to use. If not set, this is ghcr.io/kube-vip/kube-vip:v0.6.4

Interface is the name of the interface where the VIP will be configured. If not set, the default interface is used.

KubeconfigPath is the kubeconfig host path to use for kube-vip. If not set, this is: - /etc/kubernetes/super-admin.conf for the bootstrap control plane node (see https://github.com/kube-vip/kube-vip/issues/684#issuecomment-1883955927) - /etc/kubernetes/admin.conf for the rest of the control plane nodes

KubeconfigPath is useful when not using the kubeadm bootstrap provider.

ManifestPath is the path on the host where the kube-vip static pod manifest will be created. If not set, this is /etc/kubernetes/manifests/kube-vip.yaml

ManifestPath is useful when not using the kubeadm bootstrap provider.

Flavor is configuration for the instance size (e.g. t3.micro, or c2-m4).

Examples:

• t3.micro – match specs of an EC2 t3.micro instance
• c2-m4 – 2 cores, 4 GB RAM

Profiles is a list of profiles to attach to the instance.

Image to use for provisioning the load balancer machine. If not set, a default image based on the load balancer type will be used.

• “oci”: ghcr.io/lxc/cluster-api-provider-incus/haproxy:v20230606-42a2262b
• “lxc”: haproxy from the default simplestreams server

Target where the load balancer machine should be provisioned, when infrastructure is a production cluster.

Can be one of:

• name: where name is the name of a cluster member.
• @name: where name is the name of a cluster group.

Target is ignored when infrastructure is single-node (e.g. for development purposes).

For more information on cluster groups, you can refer to https://linuxcontainers.org/incus/docs/main/explanation/clustering/#cluster-groups

NetworkName is the name of the network to create the load balancer.

Name is the image name or alias.

Note that Incus and Canonical LXD use incompatible image servers. To help mitigate this issue, the following image names are recognized:

For Incus:

• ubuntu:VERSION => ubuntu/VERSION/cloud from https://images.linuxcontainers.org
• debian:VERSION => debian/VERSION/cloud from https://images.linuxcontainers.org
• images:IMAGE => IMAGE from https://images.linuxcontainers.org
• capi:IMAGE => IMAGE from https://d14dnvi2l3tc5t.cloudfront.net
• capi-stg:IMAGE => IMAGE from https://djapqxqu5n2qu.cloudfront.net

For LXD:

• ubuntu:VERSION => VERSION from https://cloud-images.ubuntu.com/releases
• debian:VERSION => debian/VERSION/cloud from https://images.lxd.canonical.com
• images:IMAGE => IMAGE from https://images.lxd.canonical.com
• capi:IMAGE => IMAGE from https://d14dnvi2l3tc5t.cloudfront.net
• capi-stg:IMAGE => IMAGE from https://djapqxqu5n2qu.cloudfront.net

Any instances of VERSION in the image name will be replaced with the machine version. For example, to use debian based kubeadm images, you can set image name to “capi:kubeadm/VERSION/debian”

Fingerprint is the image fingerprint.

Server is the remote server, e.g. “https://images.linuxcontainers.org”

Protocol is the protocol to use for fetching the image, e.g. “simplestreams”.

ProviderID is the container name in ProviderID format (lxc:///).

InstanceType is container or virtual-machine. Empty defaults to container.

InstanceType may also be set to kind, in which case OCI containers using the kindest/node images will be created. This requires server extensions: instance_oci, instance_oci_entrypoint.

Flavor is configuration for the instance size (e.g. t3.micro, or c2-m4).

Examples:

• t3.micro – match specs of an EC2 t3.micro instance
• c2-m4 – 2 cores, 4 GB RAM

Profiles is a list of profiles to attach to the instance.

Devices allows overriding the configuration of the instance disk or network.

Device configuration must be formatted using the syntax “,=”.

For example, to specify a different network for an instance, you can use:

# override device "eth0", to be of type "nic" and use network "my-network"
devices:
- eth0,type=nic,network=my-network

Config allows overriding instance configuration keys.

Note that the provider will always set the following configuration keys:

• cloud-init.user-data: cloud-init config data
• user.cluster-name: name of owning cluster
• user.cluster-namespace: namespace of owning cluster
• user.cluster-role: instance role (e.g. control-plane, worker)
• user.machine-name: name of machine (should match instance hostname)

See https://linuxcontainers.org/incus/docs/main/reference/instance_options/#instance-options for details.

Image to use for provisioning the machine. If not set, a kubeadm image from the default upstream simplestreams source will be used, based on the version of the machine.

Note that the default source does not support images for all Kubernetes versions, refer to the documentation for more details on which versions are supported and how to build a base image for any version.

Target where the machine should be provisioned, when infrastructure is a production cluster.

Can be one of:

• name: where name is the name of a cluster member.
• @name: where name is the name of a cluster group.

Target is ignored when infrastructure is single-node (e.g. for development purposes).

For more information on cluster groups, you can refer to https://linuxcontainers.org/incus/docs/main/explanation/clustering/#cluster-groups

Ready denotes that the LXC machine is ready.

LoadBalancerConfigured will be set to true once for each control plane node, after the load balancer instance is reconfigured.

Addresses is the list of addresses of the LXC machine.

Conditions defines current service state of the LXCMachine.

V1Beta2 groups all status fields that will be added in LXCMachine’s status with the v1beta2 version.

Standard object’s metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

Spec is the specification of the desired behavior of the machine.

# override device "eth0", to be of type "nic" and use network "my-network"
devices:
- eth0,type=nic,network=my-network

conditions represents the observations of a LXCMachine’s current state. Known condition types are Ready, InstanceProvisioned, Deleting, Paused.

Name is the name of the secret to use. The secret must already exist in the same namespace as the parent object.