Kubernetes example

In this example you’ll spin up a local Kubernetes cluster with NativeLink and run some Bazel builds against it.

Requirements

• An x86_64-linux system. Either “real” Linux or WSL2.
• A functional local Docker setup.
• A recent version of Nix with flake support, for instance installed via the next-gen Nix installer.

Caution

This example doesn’t work on Mac yet.

Note

On modern systems you might need to enable some backwards-compatibility kernel modules.

For instance, on NixOS:

boot.kernelModules = [
  "iptable_filter"
  "iptable_mangle"
  "iptable_nat"
  "iptable_raw"
  "xt_socket"
];

☁️ Prepare the cluster

First, enter the NativeLink development environment:

git clone https://github.com/TraceMachina/nativelink && \
    cd nativelink && \
    nix develop

This environment contains Bazel and some cloud tooling, so you don’t need to set up any kubernetes-related software yourself.

Now, start the development cluster:

native up

Tip

The native up command uses Pulumi under the hood. You can view and delete the stack with pulumi stack and pulumi destroy. If you’re queried for a stack password, press enter, as the password is an empty string.

Next, deploy NativeLink to the cluster:

kubectl apply -k \
    https://github.com/TraceMachina/nativelink//deploy/kubernetes-example

Once the infra is ready, trigger the pipelines that build the images:

cat > nativelink-repo.yaml << EOF
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: nativelink
  namespace: default
spec:
  interval: 2m
  url: https://github.com/TraceMachina/nativelink
  ref:
    branch: main
EOF
kubectl apply -f nativelink-repo.yaml

Danger

This example is built for demo purposes only. It’s not a secure production-grade setup and will only work in the local development cluster created with native up.

One-liner production-grade setups are still under construction.

🔭 Explore deployments

The deployment might take a wile to boot up. You can monitor progress via the dashboards that come with the development cluster:

• localhost:8080: Cilium’s Hubble UI to view the cluster topology. NativeLink will be deployed into the default namespace.
• localhost:8081: The Tekton Dashboard to view the progress of the in-cluster pipelines. You’ll find the pipelines under the PipelineRuns tab.
• localhost:9000: The Capacitor Dashboard to view Flux Kustomizations. You can view NatieLink’s logs here once it’s fully deployed.

In terminals, the following commands can be helpful to view deployment progress:

• tkn pr logs -n ci -f to view the logs of a PipelineRun in the terminal.
• flux get all -A to view the state of the NativeLink deployments.

Once NativeLink is deployed:

• kubectl logs deploy/nativelink-cas for the CAS (cache) logs.
• kubectl logs deploy/nativelink-scheduler for the scheduler logs.
• kubectl logs deploy/nativelink-worker for the worker logs.

🏗️ Build against NativeLink

The demo setup creates gateways to expose the cas and scheduler deployments via your local docker network. You can pass the Gateway addresses to Bazel invocations to make builds run against the cluster:

NATIVELINK=$(kubectl get gtw nativelink-gateway -o=jsonpath='{.status.addresses[0].value}')

echo "NativeLink IP: $NATIVELINK"

bazel build \
    --remote_cache=grpc://$NATIVELINK \
    --remote_executor=grpc://$NATIVELINK \
    //local-remote-execution/examples:lre-cc

Caution

While the Dashboard ports are static, the NativeLink endpoints aren’t (yet). If you shut down the cluster and reboot it, the $NATIVELINK IP addresses will change.

Tip

You can add these flags to a to a .bazelrc.user file in the workspace root. Note that you’ll need to pass in explicit IP addresses as this file can’t resolve environment variables:

.bazelrc.user

build --remote_cache=grpc://172.20.255.4
build --remote_executor=grpc://172.20.255.4

.bazelrc

try-import %workspace%/.bazelrc.user

The crucial part is this bit:

INFO: 11 processes: 9 internal, 2 remote.

It tells us that the compilation ran against the cluster. Let’s clean the Bazel cache and run the build again:

bazel clean && bazel build \
    --remote_cache=grpc://$CACHE \
    --remote_executor=grpc://$SCHEDULER \
    //local-remote-execution/examples:lre-cc

The build now shows cache hits instead of remote actions:

INFO: 11 processes: 2 remote cache hit, 9 internal.

🚀 Bonus: Local Remote Execution

The worker deployment in this example leverages Local Remote Execution.

Local Remote Execution mirrors toolchains for remote execution in your local development environment. This lets you reuse build artifacts with virtually perfect cache hit rate across different repositories, developers, and CI.

To test LRE in the cluster, clean the local cache and invoke another build against the cluster, but this time omit the remote_executor flag. This way you’ll use remote caching without remote execution:

bazel clean && bazel build \
    --remote_cache=grpc://$CACHE \
    //local-remote-execution/examples:lre-cc

You’ll get remote cache hits as if your local machine was a nativelink-worker:

INFO: 11 processes: 2 remote cache hit, 9 internal.

🧹 Clean up

When you’re done testing, delete the cluster:

# Delete the kind cluster
native down

# Remove the container registry and loadbalancer
docker container stop kind-registry | xargs docker rm
docker container stop kind-loadbalancer | xargs docker rm