Overnight reports now finish before the coffee is cold.
Engine telemetry moved from a 2009 SQL Server database to a lakehouse on S3: one data store serving engineers and airline customers.
The problem
“Every engine is streaming telemetry into a MSSQL database from 2009. Our analysts run reports overnight and pray.”
Aerospace OEM
What was at stake
- The engineers’ dashboards and the airlines’ fleet health reports ran on two forks of the same SQL Server database.
- The two forks drifted apart weekly.
- Reconciliation was a person’s Monday morning, and nobody trusted either copy.
What we built
Engineers and airlines read from the same pipeline.
S3 is the source of truth, and Redshift serves engineers and airlines from the same layer.
Before · one 2009 database, two drifting copies
Enginesstreaming telemetry
SQL Serverdatabase from 2009
Engineering dashboardsreports run overnight
Airline fleet health reportsreports run overnight
The two forks drifted apart weekly. Reconciliation was a person’s Monday morning.
After · one lakehouse, one pipeline
Enginesstreaming telemetry
Kinesis + Lambdaevent-driven ingestion
Batch reconciliationbackfill and historical replay
S3 lakehouse · source of truth
Glue and PySpark jobs move data up the layers
Bronzeraw telemetry
Silvercleaned events
Goldaggregates both audiences query
Amazon Redshiftanalytics layer
EngineersPower BI dashboards
Airline customersAPI on the Redshift endpoint
Runs on
AWS foundation · 4 accounts · 2 regions
Containerized microservices on ECS and EKS
CI/CD standardized across AWS and Azure DevOps
Simplified from the production design. 3× the telemetry volume, on half the footprint.
- 01Replaced the SQL Server monolith with an S3 lakehouse, Glue, PySpark and Redshift.
- 02Event-driven telemetry ingestion on Kinesis and Lambda, with batch reconciliation for backfill and replay.
- 03Containerized microservices on ECS and EKS, with CI/CD standardized across AWS and Azure DevOps.
- 04Org structure, Terraform modules and an on-call runbook handed over by week five.
Timeline
Sixteen weeks from kickoff to cutover.
We started with the migration plan: two phases, eleven applications, zero downtime tolerance.
Five weeksWeek fiveAWS org structure, Terraform modules and a runbook written with their on-call, handed over.
LakehouseS3 bronze, silver and gold replace the SQL Server monolith.
IngestionKinesis and Lambda stream telemetry. Batch jobs handle backfill and replay.
PlatformMicroservices on ECS and EKS. CI/CD standardized across AWS and Azure DevOps.
Week sixteenThe last colo rack is powered down. Analysts keep their Power BI dashboards.
TodayEngineers on Power BI, airlines on an API. Zero production incidents since launch.
The engagement runs 2023 to the present.
Outcomes
What changed for the OEM.
Speed
80%
faster processing of engine telemetry
Engineers and airlines now get their numbers from the same pipeline.
Scale
3x
the telemetry volume, on half the footprint
Reliability
0
production incidents since launch
Stack
ECS/EKSRedshiftKinesisLambdaGluePower BI
Tell us what you're building.
We reply within one business day.
First call: 30 minutes, on a Tuesday or Thursday, with two engineers, not a sales rep. If we’re the wrong fit, we’ll say so and name a firm that is.