Case Study
Enterprise eCommerce & Data Lake on AWS
Modernizing Retail Operations for a Fortune 500 Discount Retailer Across 16,000+ Store Locations
Client Profile
Industry Retail
Competencies
AWS Retail | AWS Migration | AWS MSP
Overview
A Fortune 500 discount retailer operating ~16,000 stores and generating $25B+ in annual revenue needed two mission-critical cloud solutions built from the ground up. ClearScale designed and delivered a serverless AWS data lake that reduced daily sales ETL processing time from 8–12 hours to under 2 hours, and a production-grade eCommerce platform across six isolated AWS accounts with full DR capability in a second region. The platform has been running in production for 4+ years.
Meet Our Hero
The client is one of North America’s largest value retailers, operating thousands of retail stores across the United States and Canada. With over 200,000 employees and annual revenue exceeding $25 billion, their digital team needed to build a modern eCommerce and Loyalty platform from scratch, while their analytics team needed to eliminate a crippling daily data bottleneck affecting decision-making across the entire store network.
The Goal
The engagement had two parallel objectives.
For the analytics team: replace a slow, inflexible on-premise batch ETL process with a cloud-native data lake that could process sales transaction data from all stores within two hours of close of business — giving analysts same-day visibility into store performance, inventory signals, and promotion effectiveness.
For the digital team: design and deliver a production-grade eCommerce platform and Loyalty API on AWS, with the security isolation, DR capability, CI/CD automation, and operational runbooks needed for a Fortune 500 retailer to own and operate independently.
The Challenge
Challenge 01
Legacy ETL Bottleneck at Scale: The client’s on-premise batch processing pipeline took 8–12 hours to process daily sales data from ~16,000 stores. By the time enriched data was available to analysts, the business day had already passed. The existing infrastructure had no path to real-time processing and required continuous capital expenditure to scale.
Challenge 02
No Cloud eCommerce Foundation: The client had no AWS-hosted digital commerce capability. The eCommerce platform and Loyalty API needed to be built from scratch — with multi-account isolation, DR, PCI-DSS-compliant network segmentation, nine backend microservices, and CI/CD automation — all within a fixed project timeline.
Challenge 03
PCI-DSS and Security Requirements: The eCommerce platform processes customer payment data across a Fortune 500 retail operation. This required account-level and VPC-level isolation for cardholder data workloads, federated identity management, SIEM integration with CloudTrail, and encryption at every layer — all designed in from day one.
Challenge 04
Knowledge Transfer and Operational Independence: The client’s internal team needed to operate both platforms independently after delivery. ClearScale had to not only build the architecture but produce comprehensive runbooks, KT sessions, and Terraform-based IaC so the customer could own the infrastructure without ongoing dependency on ClearScale.
The Solution
ClearScale delivered the engagement in two phases, each with its own architecture, CI/CD pipeline, and handover package.
Step 01: Serverless Sales Data Lake & ETL Pipeline
ClearScale designed a cloud-native ETL pipeline ingesting daily sales transaction data from ~16,000 stores via AWS Direct Connect. AWS Glue runs a four-stage pipeline — Archive → Staging → Raw → Enriched (Parquet) — with Amazon EMR handling complex Spark transformations and Kinesis Firehose covering real-time streaming paths. Glue Bookmarks enable incremental processing so only new data is processed on each run. All infrastructure is deployed via Terraform with Jenkins CI/CD — the production job requires manual approval to prevent accidental deployments. SAML federation provides scoped IAM roles, and a third-party SIEM is integrated with CloudTrail for continuous security monitoring.
Step 03: IaC, CI/CD, and Operational Handover
All infrastructure across both phases is Terraform-managed — no manual console provisioning. ClearScale delivered ten operational runbooks covering every infrastructure and application component. Knowledge transfer sessions were conducted before handover, and the customer team took full operational ownership of both platforms at project close.
Step 02: Multi-Account eCommerce Platform
ClearScale designed and built a six-account AWS architecture — Production, DR, Test, Dev, Stage, and Shared Networking — providing hard account-level isolation for cardholder data workloads. The Production VPC spans three AZs with all application and database tiers in private subnets behind a Network Load Balancer. Seven eCommerce microservices run on Elastic Beanstalk behind Amazon API Gateway. A MongoDB 3-node replica set handles the primary database, with daily automated snapshots copied to the DR region via CloudWatch Events and SSM. EFS cross-region sync runs continuously via a dedicated HA-Proxy instance. VPC peering connects Production to DR for direct failover connectivity.
The Impact
ETL processing time cut from 8–12 hours to under 2 hours, giving the analytics team same-day visibility into sales data across ~16,000 stores.
Full DR capability with automated daily database snapshots replicated to a second region, cross-region file system sync, and direct VPC failover connectivity — RTO < 4 hours, RPO < 15 minutes for transactional data.
Replacing zero cloud footprint with a fully production-grade, multi-region AWS architecture.
Turn Cloud Chaos Into Clear Results On AWS
Clearscale helps marketing and SaaS companies cut through cloud chaos and get clear results on AWS. If your legacy infrastructure is holding you back and you need a partner to tackle complex, large-scale migration and modernization projects, let’s talk.