Kubernetes Wake-up Operator

Author: MacroNova

December 10, 2018

The majority of digital companies plan nowadays their cloud journey. The triggering factor is the promise of reduced cost of IT infrastructure ownership and day-to-day maintenance, backed up by nearly infinite scaling possibilities. To fully laverage cloud flexibility, applications need to scale dynamically adapting to real-time traffic. Companies tend to improve mission ciritcal, high volume applications enabling them to scale automatically. They tend to forget about low hanging fruit to reduce overall AWS bill - components that run sporadically, for example once per week.

Let us imagine a reporting process which is triggered by JMS message. On average CEO requests hereby report once per month. It would be a costly waste of resources to keep the container running in 24/7 mode. MacroNova team developed Kubernetes Wake-up operator which can start or stop any container when given event occurs. Currently we support trigger of TIBCO EMS message, but Apache Kafka is next on the roadmap!

Below diagram presents sample JMS service deployed in Kubernetes cluster. Mentioned service listens on EMS queue queue.sample for requests. JMS client has been configured to use client acknowledge mode not to loose any not successfully processed message.

Sample JMS service deployed in Kubernetes.
Figure 1: Sample JMS service deployed in Kubernetes.

First step would be to deploy Wake-up Operator.

$ kubectl apply -f operator.yaml

Listing 1: Deploy Wake-up Operator.

Since Wake-up operator will establish new connection to EMS server, let us configure new account with limited permissions. Provided user needs to be able to view server details (connect only to active instance from fault-tolerant pair), view destination statistics, and optionally list topic durable subscriptions. Connect using tibemsadmin64 tool to EMS server and execute below commands.

tcp://localhost:7222> create user wakeup-operator password=secret
tcp://localhost:7222> grant admin user=wakeup-operator view-server,view-destination,view-durable

Listing 2: Create dedicated EMS user for Wake-up Operator.

Finally, we need to create custom Kubernetes resource that will tell Wake-up operator which container to start (or stop) upon message on given EMS destination. Of course single operator deployment can manage multiple applications, connected to different EMS instances and listening on dedicated queues.

$ cat cr.yaml
apiVersion: com.supernova/v1
kind: Wakeup
metadata:
  name: sample-jms-service-wakeup
spec:
  objectName: sample-jms-service
  objectKind: Deployment
  emsTrigger:
    serverUrl: tcp://ems-server1:7222,tcp://ems-server2:7222
    username: wakeup-operator
    password: secret
    destinationNames: [ "queue.sample" ]

$ kubectl apply -f cr.yaml
wakeup.com.supernova/sample-jms-service-wakeup created

Listing 3: Deploy custom resource to manage sample JMS service.

After a minute you should see that sample JMS service has been scaled down, because there was no message pending on queue.sample.

$ kubectl logs -l name=wakeup-operator
2018-11-24 17:43:42 INFO  WakeupOperator:53 - Scaling application [sample-jms-service] down

$ kubectl get deployments
NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
hello-minikube       1/1     1            1           5d20h
sample-jms-service   0/0     0            0           31m
wakeup-operator      1/1     1            1           19m

Listing 4: Operator terminating idle application.

Next, let us send dummy message to EMS queue using GEMS utility. Wait for a minute or so and you should see that container with sample JMS service has been started.

$ kubectl logs -l name=wakeup-operator
2018-11-24 17:43:42 INFO  WakeupOperator:53 - Scaling application [sample-jms-service] down
2018-11-24 17:47:42 INFO  WakeupOperator:53 - Scaling application [sample-jms-service] up

$ kubectl get deployments
NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
hello-minikube       1/1     1            1           5d20h
sample-jms-service   1/1     1            1           33m
wakeup-operator      1/1     1            1           22m

Listing 5: Operator starting the container.

Once the message is successfully processed and acknowledged, Wake-up operator scales down the service again.

$ kubectl logs -l name=wakeup-operator
2018-11-24 17:43:42 INFO  WakeupOperator:53 - Scaling application [sample-jms-service] down
2018-11-24 17:47:42 INFO  WakeupOperator:53 - Scaling application [sample-jms-service] up
2018-11-24 17:49:42 INFO  WakeupOperator:53 - Scaling application [sample-jms-service] down

$kubectl get deployments
NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
hello-minikube       1/1     1            1           5d20h
sample-jms-service   0/0     0            0           35m
wakeup-operator      1/1     1            1           23m

Listing 6: Operator terminating idle application.

Full reduction of infrastructure costs requires not only scaling down idle applications, but also setup of K8s auto-scaler which is part of standard Kubernetes installation. Utility removes unused nodes from the cluster and therefore releases computing resources back to cloud provider. Auto-scaler is also capable of adding servers if new pod cannot fit on any node in the cluster.

If you would like to try beta preview of Wake-up operator, do not hesitate to contact us at office@macronova.io.