In part 1, we’ve looked at how the core of Kubernetes architecture can extend beyond container orchestration, some potential use-cases as to why you would want to interact with the Kubernetes API server, and looked at different tools and libraries you can use to do so. Today we’ll be looking at how we can interact with the API server, how the APIs are structured and organized, and the kind of features it exposes. Much of the interaction will be done with the raw REST API endpoints to give you the necessary foundation in order to use higher-level .NET library abstractions that will be explored in part 3.

Hands on

I always like to experiment with this kind of stuff after I read about it, so I encourage you to try it out yourself to play with the API server. Since authentication may make it tricky to do this in cURL or Postman, but you can use kubectl with --raw the parameter to run raw queries like this:

Kubernetes API is probably best thought of as a RESTful database, exposed via HTTP and optionally WebSockets. It has a concept of object type (you can think of it as a table), and object instance (row in a database). Most (but not all) endpoints in this API follow standard convention.

https://<api-server>/apis/<group>/<version>/namespaces/<namespace>/<resource-pluralname>/<name>

I’m going to assume you’re familar with SQL server, so I’m going to make analogies to help. Lets break it down:

• api-server – the URL of the server. This would be a connection string to the database.
• group – the logical grouping under which a given resource type lives. This is equivalent to a Schema in that database which groups related tables.
• version – as resource structure evolves, we’ll need to alter schema. Kubernetes requires us to specify which version of the schema structure we’re using. This would be kind of like table having multiple updatable views, where each view is a “version” of a given a table’s schema that was made public. Only one version is actually used for storage, and there are conversion mechanics to upcast/downcast versions (more on this later).
• namespace – this lets us group objects into logical “tenants”. If we specify a namespace, only those objects that are assigned this namespace will be returned by the query. You can also omit namespace/<namespace> path component to do a cluster-wide query. This would be equivalent to every table having a column called “namespace” and including this as a WHERE clause in every query (or omitting it for a query across all tenants).
• resource-pluralname – resources have different names assigned to them depending on the context used. The plural name portion is used as the path segment to identify the resource type we’re requesting. This is equivalent to the table name.
• name – this segment is optional and is used to retrieve a single record by its unique name. This is equivalent to appending WHERE Name=<name> to a SQL query.

There’s one extra caveat. Due to legacy reasons, the concept of group did not exist in the early version of Kubernetes. Some object types (such as Pod) do not have a group. Such objects follow naming convention as follows:

https://<api-server>/api/<version>/namespaces/<namespace>/<resource-pluralname>/<name>

Data structure

The format of the actual content of each resource sent or received from the API server is actually one you’ve probably already seen if you’ve ever issued a kubectl apply. Each resource has an expected data structure to it (schema), and each resource instance needs to be formatted in accordance with the expected schema. Most Kubernetes resources follow a common convention on how they are structured. Let’s take a look at a typical Kubernetes object, specifically at highlighted lines

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/limit-ranger: 'LimitRanger plugin set: cpu request for container
      nginx'
  creationTimestamp: "2020-05-07T22:34:19Z"
  generateName: nginx2-f75d65985-
  labels:
    app: nginx2
    pod-template-hash: f75d65985
  name: nginx2-f75d65985-jn749
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: nginx2-f75d65985
    uid: b01abe47-fd99-4222-9adb-8d663fc88548
  resourceVersion: "44901864"
  selfLink: /api/v1/namespaces/default/pods/nginx2-f75d65985-jn749
  uid: 456585d9-0b4c-4a53-8a26-446a91a53c39
spec:
  containers:
  - image: nginx
    imagePullPolicy: Always
    name: nginx
    resources:
      requests:
        cpu: 100m
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-sfmwn
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: gke-play-cluster-default-pool-627cd9eb-f0h7
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: default-token-sfmwn
    secret:
      defaultMode: 420
      secretName: default-token-sfmwn
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2020-05-07T22:34:19Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2020-05-07T22:34:22Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2020-05-07T22:34:22Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2020-05-07T22:34:19Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://33aa0ff4cf4011f7802c69bb6489f60e9aed2af67ea2ec77cdb32accfdc0af90
    image: nginx:latest
    imageID: docker-pullable://nginx@sha256:cccef6d6bdea671c394956e24b0d0c44cd82dbe83f543a47fdc790fadea48422
    lastState: {}
    name: nginx
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: "2020-05-07T22:34:21Z"
  hostIP: 10.128.0.10
  phase: Running
  podIP: 10.32.0.11
  qosClass: Burstable
  startTime: "2020-05-07T22:34:19Z"

Every object will have at apiVersion and kind specified. These two fields should be treated as part of type metadata – think of kind as the equivalent of CLR type. In fact, if you’ve ever serialized using JSON.NET, one of the options is to include CLR type information in _type field – this is the same concept. apiVersion is a little different and it specifies the version of the object schema we’re targeting. Just as you would over-time change the schema of tables in the database to adapt to new requirements, this allows us to specify which version of the table structure we’re targeting. Depending on the version of Kubernetes you have installed, the API server will expect objects to be sent in one of the versions that it support (it may accept more than one version and do internal upcasting/downcasting, though it does have a single preferred version). The apiVersion specified in an object takes the form of <group>/<version> of the underlying resource. You can find which versions the API server supports via kubectl api-versions.

Almost all objects will also have a property called metadata which has the same structure in all objects. A few important fields in particular to note in metadata:

• name – this is a unique identifier per resource type, and for the most part, can be thought of as primary key. This is also what is used as part of REST path for name segment.
• uid – randomly generated identifier ID. The difference between uid and name is that if an object is deleted, another one can be recreated with same name. uid is never reused.
• namespace – the namespace object belongs to. this can be blank if the object is created at cluster level
• labels – a key/value dictionary applied to object which for the most part is equivalent to tags that can have values. This is one of main ways to search for objects.
• annotations – similar to labels, except non-searchable. This is just a way to attach some helpful metadata about the object
• resourceVersion – specifies the revision of the object which can be used to detect if the resource was updated while being disconnected from the server. This field is automatically updated by the server any time an object is updated. While this field should be treated as opaque token, it can be thought of as the timestamp when a resource was last modified and can be used for queries such as “give me all resources modified after X timestamp”.
• ownerReferences – if an object is inherently created and linked to another (parent-child types relationship), you can use ownerReferences to track this. This also allows you to apply cascade delete policy to automatically delete child objects if a parent is deleted.

Restful API

Being a RESTful API, we can use standard HTTP verbs to query (GET), create (POST), or full replace (PUT) or partial update (PATCH). Kubernetes, however, uses its own terms to classify the types of operations you can issue. Lets briefly discuss them

• Get– an HTTP GET request for single resource where name is provided as port of URL path segment. ex. api/v1/pods/mypod. This returns a single response of serialized version of resource (kind:Pod)
• List– a request that can return multiple resources of same type. This is an HTTP GET request without a name segment (ex. api/v1/pods).

• Create – create a new resource. This is done via HTTP POST to LIST endpoint
• Update – “replace” operation to an existing resource that takes full resource version and replaces it. It is an HTTP PUT operation is done to the same endpoint a Kubernetes Get operation
• Patch – this allows partial update to an existing resource, allowing only deltas of what changed to be sent and applied server-side. The payload is expected to be of either JSONPatch or JSON Merge Patch and is done via HTTP PATCH.
• Delete – this deletes a single resource. This is an HTTP DELETE request to Get endpoint.
• DeleteCollection – this deletes multiple resources that match the result of a particular query type. This is done via HTTP DELETE to List endpoint. You can use the same filtering capabilities as for List
• Watch – allows monitoring a request for changes. This type of request uses a never-ending HTTP request or WebSocket to get the current state of a resource query and observe the changes going forward. This type of request is very powerful and what makes Kubernetes API very efficient as you’re able to monitor and react to changes in the Kubernetes database in near real-time without the need for polling. A Watch request is issued as a HTTP GET against List endpoint with query string parameter of watch=true. (In older version of Kubernetes, watch requested used a separate URL such as /api/v1/watch/pods. This is now deprecated in favor of query-string parameter).

Bookmark is used to update resourceVersion and is primarily intended to optimize certain types of recovery scenarios when disconnected from server.

Querying

If we wanted to list resources, we can issue a GET operation to /api/<version>/<resource-pluralname>. We’ve already discussed the use of namespace and name path segments as some of the filtering capabilities. You can further narrow down the search of the query via query-string parameters:

• labelSelector – search for resources with specific labels. You can specify more than one via simple selector syntax. You can get more info on this on official Kubernetes docs
• fieldSelector – filter resources on the values of fields in the resource body. You can use JSONPath to select a field with an equality operator (ex. metadata.namespace=default). Unfortunately, this is not a generic filter for any arbitrary field in the object. Only some fields are supported per resource type, and the supported fields are not well documented. Since all objects have metadata, metadata.namespace and metadata.name are available common field selector.
• resourceVersion – when used as a query string parameter, this will give us resources that have been updated AFTER the value provided by this parameter.

Query Responses

List type response queries return items as part something called List response, where the Kind of the response will be <plural-name>List. For example, if you request to api/v1/pods what you will get back is

{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
    "selfLink": "/api/v1/pods",
    "resourceVersion": "46071741"
  },
  "items": [
  ... array of Pod objects
]
}

An interesting thing to note here is that List has its own resourceVersion, and it will be different each time you make a request for the list. If you think about it, this makes sense, since the list is dynamically generated (aka result of a query), its resourceVersion would be set to timestamp when it was created.

Request to Kubernetes Get operation (ex. api/v1/pods/nginx2-f75d65985-t2xt9), the response will be of type Pod, not PodList.

For watch type queries, the results are sent as new-line delimited JSON blocks of type WatchEvent data structure that looks as follows:

{
  "type": "ADDED",
  "object": {
   ... actual object ...
  }
}

The type can be either ADDED, MODIFIED, DELETED, ERROR, and BOOKMARK. In case of ERROR, the object body will actually be of type Status with details of the error, for example:

{
  "type": "ERROR",
  "object": {
    "kind": "Status",
    "apiVersion": "v1",
    "metadata": {},
    "status": "Failure",
    "message": "too old resource version: 1 (45974026)",
    "reason": "Gone",
    "code": 410
  }
}

Experiments

I encourage you to play with API server to get a feel on how it works. Some endpoints to try

• api/v1/pods
• api/v1/namespaces/default/pods
• api/v1/pods?watch=true (don’t pipe to jq as its a streaming request)
• api/v1/configmaps
• api/v1/configmaps/<name>

The following endpoints are metadata endpoints. They do not follow “standard” resource rules described above and instead used to discover which resource types and versions the server supports

• api
• api/v1
• apis
• apis/<group>
• apis/<group>/<version> (ex. apis/apiextensions.k8s.io/v1beta1)

What’s next

In next part of the series we’ll explore how to use .NET Kubernetes Client to interact with Kubernetes API server.