Three AWS-centric Projects Help Agari Maintain Email Security Leadership

Executive Summary

Agari is the Trusted Email Identity Company™, protecting brands and people from devastating phishing and socially-engineered attacks. Over several months, ClearScale partnered with Agari on three diverse projects — all requiring extensive AWS expertise in several areas ranging from staff augmentation to database migration.

The collaboration enabled Agari to maintain its aggressive pace of product and service releases and updates — which is critical for maintaining its position as the email security category leader.

Project 1: Staff Augmentation

ClearScale was initially enlisted to provide Agari with short-term staff augmentation services in the areas of AWS ElasticSearch and AWS ElastiCache for Redis.

For ElasticSearch, the services entailed application architecture and development services and included the review and analyses of Agari’s data pipelines, caching strategy, and more. For AWS ElastiCache, ClearScale reviewed and analyzed slow logs, searching options and queries, Agari’s data rotation strategy, and other components.

ClearScale was also responsible for numerous DevOps services including infrastructure architecture and various tasks related to Amazon RDS, DynamoDB, ElastiCache Redis and ElasticSearch.

All requests from Agari were input into Jira, a project tracking system. ClearScale successfully executed them based on priority and urgency. Detailed reports were delivered for all completed tasks.

Project 2: Sensor Monitoring

Pleased with ClearScale’s deep expertise and on-time delivery, Agari next tasked the company with developing a monitoring solution to gather health information for its secure email gateway hosted sensors that help intercept malicious messages and block or redirect them to quarantine for further incident investigation. The project also required a monitoring dashboard to improve metrics visibility and filtering.

The ClearScale team designed, tested and deployed a solution that uses Prometheus, an open source monitoring system for metrics aggregation, and Grafana Lab’s Grafana, an open source platform for visualizing metrics.

A highly available configuration of Prometheus was installed in a cluster with Amazon Elastic Kubernetes Service (Amazon EKS). Prometheus’ AlertManager communicates alerts to PagerDuty, an incident management platform. Persistent storage — the storage volumes that remain available beyond the life of individual containers — was used for historic data from Prometheus. Amazon Aurora was integrated with Grafana to build the dashboards for both every day and special use.

In addition, the ClearScale team delivered automation tools to setup and monitor the solution, along with documentation. Now, all newly created sensors can be set up with monitoring and registered in the monitoring system.

After implementing the new monitoring solution, Agari has been able to grow its network of sensors worry-free and at a much lower cost compared to other solutions.

Project 3: Database Migration

Agari then entrusted ClearScale with yet another project: this one requiring a proof of concept (POC) for migrating two self-hosted databases (on AWS EC2 instances) to AWS RDS/ PostgreSQL and Aurora.

The project presented a number of challenges. One of the databases was PostgreSQL 9.3 and the other was Citus 6.0 on PostgreSQL 9.5. Neither PostgreSQL version is still being supported.

Citus had primarily been used for distributed queries and batch loading. With a new version released, the transition would be much more cumbersome. In addition, no more than one hour of downtime was acceptable for the migration process. Performance had to be maintained at the same level.

ClearScale’s solution entailed using Amazon Relational Database Service (Amazon RDS) with one master and two slaves using c5.xlarge instances for the Postgres 9.3 database. Two three-node Aurora clusters were configured to host the Citus data.

Data migration was performed using Bucardo, an asynchronous, multi-master, master-slave table-based replication system.

ClearScale reviewed the current data schema and architecture, created a transition plan, defined migration tools, conducted a POC of the migration, and assisted with query performance tests.

Following the POC, Agari was able to plan the migration of its production databases to a managed AWS offering. The result: reduced environment complexity and less administrative burden required to manage the self-hosted databases.

General Results

ClearScale supplied the resources and expertise Agari required, freeing up its internal staff and enabling the implementation of three successful projects.

All project requirements were met and Agari’s expectations exceeded, thanks to ClearScale’s focus on collaboration and open communication, its AWS expertise, and the on-time delivery of its well thought out solutions.

Specifically for the sensor monitoring project, the custom-built monitoring system provides a versatile means of tracking health and performance of hundreds of hosted sensors. This eliminates costly licensing and adds flexibility.

For the database migration project, with databases migrated to Amazon managed offerings like Aurora and RDS for PostgreSQL, Agari is able to keep its infrastructure secure and up to date with minimal effort. At the same time, it’s able to maintain high performance and availability of core data storage.

Perhaps most important, Agari gained a value-added and trusted partner that it can count on for future projects.