Migrating from AWS Lambda to EKS: Our Journey and Key Learnings
At Trestle, we initially built our product APIs on AWS Lambda, leveraging its serverless capabilities to scale dynamically. However, as our product grew and low-latency requirements became critical, we encountered several challenges that prompted us to migrate to Kubernetes (EKS). This blog details our migration journey, the challenges we faced with Lambda, our architectural decisions, and the impact of moving to Kubernetes (EKS) and Quarkus.
Why We Started With AWS Lambda?
AWS Lambda is a serverless computing service that executes code responding to events without provisioning or managing servers. It scales automatically and only charges for execution time, making it an attractive choice for running lightweight microservices.
Concurrency in AWS Lambda
Lambda’s concurrency defines how many requests can be handled in parallel. It includes:
| Concurrency Type | Definition |
| Provisioned Concurrency | Keeps a predefined number of Lambda instances warm to reduce cold starts. |
| Reserved Concurrency | Limits the maximum concurrent executions to ensure other functions don’t consume all available capacity. |
Challenges Faced with AWS Lambda
While Lambda worked well initially, we encountered two major challenges:
1. Cold Start & Scalability Issues
- Lambda functions initialize when invoked, leading to latency issues for low-latency APIs.
- Even with Provisioned Concurrency, the cost of keeping many instances warm was high.
- Uneven traffic patterns caused delays in initialization, making it difficult to ensure consistently low latency.
Example Scenario
- Suppose a Lambda function starts with one concurrent execution, and a new user with a 10 QPS plan makes requests.
- Ten new Lambda instances must initialize, taking ~five seconds for DB connections and dependencies.
- Each request in that batch now experiences five seconds + actual processing time, making it unsuitable for low-latency APIs.
The below figure shows the max time, which increases to seven to eight seconds whenever there is a fluctuation in the traffic and requires the new lambdas to initialize in order to serve those requests.
2. Handling Background Jobs Efficiently
- We needed to extract insights from each request and push them to our feedback network service.
- In Lambda, the request had to wait until this process was complete, impacting latency.
- Running background jobs in Lambda was unreliable because instances could freeze or be terminated before completing async tasks.
Migrating to Kubernetes (EKS)
To address these challenges, we migrated from AWS Lambda to EKS, which allowed better resource utilization, reduced latency, and more control over scalability.
Why EKS Over Lambda?
| Feature | AWS Lambda | EKS (Kubernetes) |
| Concurrency | One Lambda serves one request at a time. | One pod can handle multiple requests. |
| Cold Starts | High latency during initialization. | Pods remain warm and handle requests instantly. |
| Auto-scaling | Scales based on invocation frequency. | Scales based on CPU & memory usage. |
| Background Jobs | Unreliable due to function freezing. | Can run persistent background jobs. |
| Latency Sensitivity | High under sudden traffic spikes. | Can handle burst traffic efficiently. |
How Kubernetes Solved Our Issues
- Better Request Handling: Unlike Lambda, where one instance serves only one request, Kubernetes allows a single pod to handle multiple concurrent requests.
2. Efficient Scaling: One pod could be allocated sufficient CPU and memory to handle traffic spikes without the cold start issues seen in Lambda.
3. Persistent Background Jobs: Since pods remain active, we could run background threads reliably without worrying about them being abruptly terminated.
Code Migration From Lambda to EKS
We initially wrote our Lambda-based code in plain Java. As part of the migration, we converted it to Spring Boot for structured REST API development and then moved to Quarkus for better performance.
Key Code Changes
| Change | Reason |
| Converted plain Java to Spring Boot | Standardized REST API development. |
| Handled concurrency in the code | Lambda was single-threaded, while Kubernetes required handling race conditions. |
| Optimized DB & API connection pools | Lambda handled one request at a time, while EKS required proper connection pooling. |
| Tuned CPU & Memory Allocation | Ensured efficient resource usage for optimal performance. |
Resource Allocation in Kubernetes
Proper resource allocation was crucial for maintaining performance.
| Resource | Description |
| CPU Allocation | Needs to be optimized—too high and it’s inefficient; too low and initialization times increase. |
| Memory Allocation | Allocating too little risks Out of Memory (OOM) errors; too much leads to wasted resources. |
| Requests and Limits | Defines the minimum and maximum CPU & memory usage a pod can consume. |
Our Case with Resources:
- 800m CPU and 512MB RAM → Served 4 requests per pod with 30s initialization time.
- Running 2 pods handled 8 requests using a 1600m of 2000m CPU machine.
- Adding a 3rd pod required a new machine, leading to delays due to node provisioning.
- The bottleneck was node provisioning:
- Stabilization time: ~30s.
- New machine provisioning (AWS side): ~2-3 minutes (uncontrollable).
- Pod startup time: ~30s.
- Total time until system stabilized: ~3-4 min.
Optimizations Tried:
- Adjusted stabilization time (tried 15s → too low, spiky scaling; tried 45s → too long, delayed new pod arrivals). We eventually settled for 30s as the optimum number for our use case and traffic requirements.
Key Learnings on Kubernetes Scaling and Deployments
- Anti-Entropy in Pod Scheduling
- Initially, all pods were running on the same node.
- New pods are scheduled on a different node when rolling out new code, causing service disruptions.
- We enforced anti-affinity rules to distribute pods across multiple nodes for smoother updates.
- Blue-Green Deployment for Stability
- We implemented traffic transfer policies (20% → 50% → 70% → 100%) to ensure smooth rollouts.
- Key Kubernetes Probes Used:
- Readiness Probe: Ensured the new pod was fully ready before serving traffic.
- Liveness Probe: Restarted unhealthy pods to maintain availability.
- Startup Probe: Allowed extra time for slow-starting services before marking them as failed.
Adding Kubernetes Helm Configuration for Deployment
We utilized Helm to manage our Kubernetes configurations to ensure a seamless Kubernetes setup and deployment. Below is the essential Helm template others can refer to when setting up a similar Kubernetes-based microservices architecture.
- Deployment YAML (Helm Template): Defines replica count, resource allocation, probes, and affinity rules.
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "prod-eks-app.fullname" . }}
namespace: prod-eks-api
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
app.kubernetes.io/name: {{ .Chart.Name }}
template:
metadata:
labels:
app.kubernetes.io/name: {{ .Chart.Name }}
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
topologyKey: "kubernetes.io/hostname"
containers:
- name: {{ .Chart.Name }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
ports:
- containerPort: {{ .Values.service.port }}
livenessProbe: {{- toYaml .Values.livenessProbe | nindent 12 }}
readinessProbe: {{- toYaml .Values.readinessProbe | nindent 12 }}- Auto-scaling (HPA): Dynamically scales pods based on CPU utilization.
{{- if .Values.autoscaling.enabled }}
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: {{ include "prod-eks-app.fullname" . }}
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: {{ include "prod-eks-app.fullname" . }}
minReplicas: {{ .Values.autoscaling.minReplicas }}
maxReplicas: {{ .Values.autoscaling.maxReplicas }}
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
{{- end }}- Canary Deployment (Argo Rollouts): Manages progressive rollouts to prevent downtime.
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: {{ include "prod-eks-app.fullname" . }}
spec:
strategy:
canary:
canaryService: {{ include "prod-eks-app.fullname" . }}-canary
stableService: {{ include "prod-eks-app.fullname" . }}
steps:
- setWeight: 20
- pause: {duration: 10s}
- setWeight: 50
- pause: {duration: 10s}
- setWeight: 100- Service: Defines how the application is exposed inside Kubernetes.
apiVersion: v1
kind: Service
metadata:
name: {{ include "prod-eks-app.fullname" . }}
spec:
selector:
app.kubernetes.io/name: {{ .Chart.Name }}
ports:
- protocol: TCP
port: {{ .Values.service.port }}
targetPort: {{ .Values.service.port }}
type: ClusterIP- Helm Values File (values.yaml): Customizable settings for replicas, autoscaling, probes, and resources.
replicaCount: 2
image:
repository: myrepo/prod-eks-app
tag: latest
autoscaling:
enabled: true
minReplicas: 2
maxReplicas: 5
targetCPUUtilizationPercentage: 80
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 10
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 5Exploring Quarkus for Better Performance
After stabilizing our Kubernetes setup, we explored Quarkus to further optimize resource utilization.
Why Quarkus?
Quarkus is a Kubernetes-native Java framework optimized for low memory usage and fast startup times.
| Feature | Spring Boot | Quarkus |
| Startup Time | Slower (~30s) | Faster (~15s) |
| CPU Usage | 800m served 4 requests with no queueing for context switching | 800m served 8 requests with no queueing for context switching |
| Memory Usage | Higher | Lower |
| Performance | Moderate | Optimized for cloud & serverless |
| Integration | More libraries & community support | Lightweight & container-friendly |
Performance Gains with Quarkus
- Requests per Pod: Doubled from four to eight without latency increase.
- Startup Time: Reduced from 30s to 15s.
- Resource Efficiency: Comparatively less CPU & RAM compared to Spring Boot.
Final Thoughts and Future Roadmap
Bottlenecks & Challenges
- Observability & APM Integration
- Spring Boot had seamless APM (Application Performance Monitoring) support via Kibana APM.
- Quarkus does not natively support running the Kibana APM agent with the JAR file, requiring a shift to telemetry-based monitoring.
- Error Logging with Sentry
- Spring Boot had direct integration with Sentry, making error tracking easy.
- Quarkus required using pure Java Sentry SDK, making integration slightly more manual.
- Documentation & Community Support
- Spring Boot has extensive resources due to its industry-wide adoption.
- Quarkus has a smaller community, making it harder to find solutions for edge cases.
Future Plans: Full GraalVM Native Migration
Our next goal is to fully migrate to GraalVM native mode, further improving startup time and resource efficiency.
To achieve this, we need to replace standard Java libraries with Quarkus-supported dependencies, such as:
- Quarkus Sentry for error tracking.
- Quarkus APM alternatives for telemetry.
- Quarkus AWS Service dependencies for cloud service integrations.
Once we complete the migration to GraalVM native mode, we expect:
- Near-instant startup times.
- Further reduction in memory usage.
- Better optimization for cloud and containerized environments.
Conclusion
Migrating from AWS Lambda to Kubernetes (EKS) significantly improved our system’s scalability, concurrency handling, and background processing capabilities. Adopting Quarkus further reduced resource consumption and startup times. Our future focus is on fully leveraging GraalVM native mode to push these optimizations even further.
