Ebates Journey to Cloud-Hosted CI/CD to Achieve a High Elasticity and Lower Time-To-Value

Executive Summary

Ebates is one of America’s most popular online cashback websites. Founded in 1999, the San Francisco-based company partners with major retailers such as Amazon, Best Buy, and Verizon to help its members earn cash back on every online purchase with no strings attached. As one of the largest free-membership loyalty programs in the U.S., Ebates has more than 10 million members and, thanks to its strong focus on offering the best possible customer experience, has earned an A+ rating from the Better Business Bureau. In 2014 Ebates was acquired by Rakuten, the largest Japanese ecommerce conglomerate, for $1 billion.

Since the company’s inception, Ebates members have earned over $1 billion in cash back. In 2017, more than $10 billion was spent using Ebates websites, accounting for 5% of all U.S. shopping traffic at the peak of its popularity that year. With a global reach of more than 1 billion users, Ebates’ profitable, high-growth business continues to expand, helping users across the globe save money on nearly every online purchase and in-store purchases at major brands.

The Challenge

Due to its sheer number of users and its rapid growth, Ebates’ fast-growing business was outpacing the capacity of its on-premise infrastructure. This was evident over the course of the typical retail sales cycle: the fourth-quarter holiday season demanded expanded capacity, while operations scaled back after the new year. Being a fast-paced company, Ebates tries out new rewards models and tests these models with real user traffic. These business trials needed multiple parallel environments that demand ephemeral infrastructure with CI/CD capabilities to enable flexible business trials. Considering that millions upon millions of Ebates users rely on the company’s sites around the clock, it was critical for Ebates to implement a new infrastructure solution that support CI/CD without disrupting existing operations.

Ebates desired new, elastic capabilities to ensure the best user experience and fastest possible loading times during the peak fourth-quarter period, while preventing the overprovisioning of IT resources during the rest of the year. Ebates wanted to achieve better cost optimization and an improved security posture. Additionally, Ebates sought enhanced infrastructure management, code deployment processes, automation, and containerization of services. This was no small task as Ebates has dozens of services within each environment supporting specific functions and localized sites.

To achieve these goals, Ebates looked to move the majority of its platform’s infrastructure from two on-premises data centers into the Kubernetes based CI/CD infrastructure in the cloud. The company desired to maintain a hybrid cloud approach, at least for some time, to be able to utilize its existing legacy hardware for Big Data and ML tasks. Ebates reached out to ClearScale, an AWS Premier Consulting Partner, to help carry out the project.

The ClearScale Solution

ClearScale worked meticulously with Ebates to create a holistic understanding of their business objectives, functional requirements, and complex systems. Ebates’ applications and environments had been developed over years of operations with increasing complexity, and although the designs were aligned with best practices, they revolved around on-premise infrastructure and were not cloud-centric. As a trusted AWS Premier Consulting Partner, ClearScale possessed the knowledge and experience to carry out an extremely large-scale migration seamlessly while closely collaborating with Ebates to create an ideal solution to satisfy their unique needs.

Leveraging proven design methodologies, ClearScale worked collaboratively with Ebates to develop the overall infrastructure design and migration strategy for each functional environment. The infrastructure design is based on the “AWS Landing Zone” concept in which a multi-account model is used to organize and manage accounts by effective role/function. The design also centralizes services to their respective functions.

Security Operations has a separate account for their core functions (CloudTrail logs, AWS Config, GuardDuty, security logs, vulnerability scanners, etc.). Services accessed by corporate users and developers (Jenkins, Confluence, Jira, Tableau, etc.) are centralized in a corporate access account. Workloads in scope of PCI compliance are also isolated by account to reduce attack surface and limit blast radius from security incidents affecting other services. Production and non-production workloads are also isolated in separate accounts, with management tools and customer facing services in different VPCs, as a best practice. All environments adhere to infrastructure-as-code and DevOps automation principals for agility, speed, control, and compliance.

The great benefit of using the AWS Landing Zone strategy is that all services can be tied back to centralized functions. Ebates is able to enforce standards for these functions, such as storing immutable logs securely, controlling resource, automating deployment, and managing costs. They can operate on a central user directory that is integrated with IAM and multi-factor authentication. Ebates has visibility and controls that elevate their security posture while realizing the business value of Hybrid Cloud.

Nuts and Bolts

After completing the high-level infrastructure design, we began the process of vetting low-level design details against Ebates’ requirements and stringent security controls. There was a lot of ground to cover: IDS/IPS, DDoS protection, service elasticity, business continuity, disaster recovery, automation, CI/CD pipelines, alerting, etc. The strategy for technology selection was to evaluate requirements against native cloud services first, and look to third-party solutions only if there were gaps. Here are two interesting examples of how unique requirements were addressed:

Perimeter Security

Ebates has a well-versed security operations team that is constantly pushing the envelope on ingress and egress controls. They develop and manage their own WAF and egress rules. False-positive blocking does occur in tightly controlled environments, so Ebates has additional integration in their application layer to allow users to submit tickets if they encounter access issues. This is an automated process and data is collected directly from the WAF service to help Ebates continuously improve security. Ebates also has strict controls for egress traffic, locking down outbound calls to known partners and vendors. For ingress and egress protection, custom solutions were implemented for the highest level of control.

AWS Elastic Container Service for Kubernetes

Several localized sites would be re-architected to run as containerized applications. The target platform was the newly released EKS service, AWS Elastic Container Service for Kubernetes. ClearScale and Ebates worked together to implement Kubernetes plugins to restrict network access within the cluster (Calico Network Policy), enable container authentication to AWS resources (kube2iam), and setup RBAC integration with IAM for the Kubernetes API (heptio). We also experimented with several logging and monitoring tools to provide operational visibility into Kubernetes. Additional service segregation was implemented by using both namespaces and separate clusters. The end result was a container platform that was compliant with Ebates’ requirements for role-based access controls and container isolation, while being Cloud-independent due to Kubernetes based container deployment built on a managed AWS service.

Migration Process

A phased migration plan was developed in order to tackle the complex task of moving Ebates’ workloads to the cloud. The transaction processing platform was the first system scheduled for migration. This PCI compliant environment was identified largely as a “rehost” type migration. Applications were to be deployed on EC2 instances with some modifications to the automation framework for enabling auto-scaling features in AWS. Security controls were deployed using their cloud-friendly versions and additional AWS services (such as GuardDuty and CloudTrail) layered on top. Due to the sensitivity of the PCI environment, all efforts were focused on this initial phase.

The next phase of migration included the corporate services environment. The corporate services included back-office services, an Atlassian stack, Tableau, Directory services, internal databases, data analysis and visualization tools, and back-end security appliances. Part of the user machine fleet was moved to AWS Workspaces. This simplified management, automating administration tasks and patch management while providing a secure remote environment.The corporate services were migrated from their stand-alone systems to true HA configurations backed by resilient AWS services (EFS, EBS, RDS, etc.) with real disaster recovery capabilities.

In parallel with the corporate services migration, iterative development was completed on EKS resulting in a production ready platform. This work was done in preparation for the migration of their regional sites. The regional sites/environments were identified as “re-architect” type migrations. The applications were to be containerized and deployed on EKS. Moving to a new platform required development of new provisioning, build and deployment workflows and automation. Ebates Canada was deployed and tested on the functional PoC.

Ebates Canada was migrated to cloud on the new EKS platform. This successful migration was a great milestone for Ebates as it validated the new framework and set the standard for the remaining sites. It served as a template for regional site migrations planned throughout 2019. Based on this proven process, Ebates had the confidence to start migration of their flagship site, Ebates.com (U.S.) to containerized environment in the cloud. The remaining phases are focused on repeating this process for all regional sites.

The Benefits

ClearScale allowed Ebates to implement fully automated, version-controlled configuration deployments and management through every phase of the deployment cycle. The result was a significant increase in speed, with the speed of a typical application deployment cycle improving by more than 50%. Designing the automation script templates to work in all regional environments allowed the regional site deployment cycle to speed up dramatically, from four to six months to just a few days. Both infrastructure and application were updated, enabling constant controlled changes and experimentation that could be rolled back in a matter of minutes.

Compared to Ebates’ previous on-premise setup, migrating several hundreds of virtual machines to EKS enabled the company to drastically decrease its overall number of virtual machines in the cluster, fully optimizing its utilization and achieving a cost savings of 60%. ClearScale’s assistance in the AWS migration enabled Ebates to surpass both of its migration goals, maintaining or significantly improving each component’s performance and security.

With ClearScale’s help, Ebates now enjoys an environment that offers higher scalability, elasticity, and security, ensuring its operations remain stable and flexible year-round, with precise “just-in-time” provisioning. Serving millions of members and processing billions of dollars in payments, Ebates’ new infrastructure can now handle the most demanding traffic loads during the busy holiday shopping season, while automatically shrinking during the off-seasons.

For future site improvements, Ebates’ improved deployment capabilities now make it easy to deploy both application and infrastructure code automatically, greatly enhancing its continuous integration and continuous delivery capabilities, and enabling Infrastructure-as-Code practice. Just as Ebates seeks to save its users money on everyday purchases, ClearScale’s AWS-powered solution will help Ebates save money on ongoing infrastructure costs.