Skip to content
Fluid Attacks Development Portal

Cluster

Cluster is the component of Common in charge of providing a company-wide Kubernetes Cluster.

Public Oath

Fluid Attacks will constantly look for new stack that simplifies serving applications to the Internet in a secure and automated manner.

Architecture

Architecture-light Architecture-dark
  1. The module is managed as code using Terraform.
  2. We have one Kubernetes cluster called common-k8s that is shared by all the components.
  3. The cluster is hosted on AWS.
  4. The cluster is divided into namespaces, which keep resources in isolation from other namespaces.
    • The default namespace is unused, we try to put things into a namespace appropriate to the product.
    • The dev namespace currently holds the ephemeral environments of Integrates.
    • The prod-integrates namespace holds the production deployment of Integrates, and a Celery jobs server.
    • The kube-system namespace holds cluster-wide deployments for, load balancer, DNS, node termination handler, cloudflared, observability and autoscaler.
    • Other kube-* namespaces exist, but they are not used for anything at the moment.
  5. Every namespace runs in a specific worker group whose physical machine instances run on EC2.
  6. The cluster spawns machines on all availability zones within us-east-1 for maximum spot availability.
  7. The cluster supports autoscaling based on several metrics like cpu consumption, memory consumption and queue size.
  8. It provides observability tools for debugging.
  9. It creates AWS application load balancers and Cloudflare DNS records for ingress resources.
  10. It supports a Cloudflare ZTNA tunnel that allows developers to access the AWS VPC via Cloudflare WARP.
  11. Developers can access the cluster via Okta for debugging.

Contributing

Please read the contributing page first.

General

  1. Any changes to the cluster infrastructure and configuration must be done via Merge Requests.
  2. Any changes related to the Platform (deployments, autoscaling, ingress…) for both development and production must be done via Merge Requests.

Components

Our cluster implements:

  1. AWS EKS Terraform module for declaring the cluster as code using Terraform.
  2. AWS Load Balancer Controller for automatically initializing AWS load balancers when declaring ingress resources.
  3. AWS Kubernetes Autoscaler for automatically scaling the cluster size based on resource assignation.
  4. ExternalDNS for automatically setting DNS records when declaring ingress resources.
  5. Kubernetes Metrics Server for automatically scaling deployments like production Platform based on application load (CPU, Memory, custom metrics).
  6. New Relic for monitoring both production Platform and general infrastructure.

Debugging

Connect to cluster

In order to connect to the Kubernetes Cluster, you must:

  1. Login as an Integrates developer using this guide.
  2. Install kubectl and aws-cli with nix-env -i awscli kubectl.
  3. Select cluster by running aws eks update-kubeconfig --name common-k8s --region us-east-1.
  4. Run kubectl get node.

Your input should be similar to this:

Terminal window
kubectl get node
NAME                            STATUS   ROLES    AGE   VERSION
ip-192-168-5-112.ec2.internal   Ready    <none>   58d   v1.17.9-eks-4c6976
ip-192-168-5-144.ec2.internal   Ready    <none>   39d   v1.17.11-eks-cfdc40
ip-192-168-5-170.ec2.internal   Ready    <none>   20d   v1.17.11-eks-cfdc40
ip-192-168-5-35.ec2.internal    Ready    <none>   30d   v1.17.11-eks-cfdc40
ip-192-168-5-51.ec2.internal    Ready    <none>   30d   v1.17.11-eks-cfdc40
ip-192-168-6-109.ec2.internal   Ready    <none>   30d   v1.17.11-eks-cfdc40
ip-192-168-6-127.ec2.internal   Ready    <none>   18d   v1.17.11-eks-cfdc40
ip-192-168-6-135.ec2.internal   Ready    <none>   31d   v1.17.11-eks-cfdc40
ip-192-168-6-151.ec2.internal   Ready    <none>   30d   v1.17.11-eks-cfdc40
ip-192-168-6-221.ec2.internal   Ready    <none>   13d   v1.17.11-eks-cfdc40
ip-192-168-7-151.ec2.internal   Ready    <none>   30d   v1.17.11-eks-cfdc40
ip-192-168-7-161.ec2.internal   Ready    <none>   33d   v1.17.11-eks-cfdc40
ip-192-168-7-214.ec2.internal   Ready    <none>   61d   v1.17.9-eks-4c6976
ip-192-168-7-48.ec2.internal    Ready    <none>   30d   v1.17.11-eks-cfdc40
ip-192-168-7-54.ec2.internal    Ready    <none>   39d   v1.17.11-eks-cfdc40

Common commands

Most commands have the following syntax: kubectl <action> <resource> -n <namespace>

  • Common actions are: get, describe, logs, exec and edit.
  • Common resources are: pod, node, deployment, ingress, hpa.
  • Common namespaces are: development, production and kube-system. Additionally, the -A flag executes <action> for all namespaces.

Some basic examples are:

CommandExampleDescription
kubectl get pod -AN/AGet all running pods
kubectl get node -AN/AGet all cluster nodes
kubectl get deployment -AN/AGet all cluster deployments
kubectl get hpa -AN/AGet all autoscaling policies
kubectl get namespaceN/AGet all cluster namespaces

Some more complex examples are:

CommandExampleDescription
kubectl describe pod -n <namespace> <pod>kubectl describe pod -n development app-dsalazaratfluid-7c485cf565-w9gwgDescribe pod configurations
kubectl logs -n <namespace> <pod> -c <container>kubectl logs -n development app-dsalazaratfluid-7c485cf565-w9gwg -c appGet container logs from a pod
kubectl exec -it -n <namespace> <pod> -c <container> -- <command>kubectl exec -it -n development app-dsalazaratfluid-7c485cf565-w9gwg -c app -- bashAccess a container within pod
kubectl edit deployment -n <namespace> <deployment>kubectl edit deployment -n development integrates-dsalazaratfluidEdit a specific deployment