Modernizing and Automating a Production Web Estate on AWS

Modernized a multi-site AWS web estate with Terraform, automated operations, recovery tooling, monitoring, and engineered rollback paths for safer production delivery.

My Role

Senior engineer responsible for infrastructure architecture, operational automation, and reliability improvements.

Problem

A fragmented web estate was difficult to maintain, with host-specific operations, manual updates, and limited recovery paths across production and archived properties.

Context / Constraints

• The estate included production and archived web properties across AWS and mixed hosting.
• Operations had relied on host-specific knowledge and manual SSH-based maintenance.
• Changes needed reproducible release behavior and recovery safety without introducing unnecessary complexity.

Architecture

• Infrastructure: AWS and Terraform for reproducible environment configuration.
• Operations: tag-based runtime discovery, AWS Systems Manager, versioned artifacts, and reversible deployment paths.

Key Engineering Decisions

• Replaced host-specific maintenance with tag-based runtime discovery and AWS Systems Manager, eliminating direct SSH from routine fleet operations.
• Used immutable image tags and infrastructure-controlled versioning to make runtime changes traceable and reversible.
• Engineered rollback paths with one-to-one Route53 forward/rollback scripts and pre-change backups.

Implementation

• Implemented Terraform-managed AWS infrastructure for the web estate.
• Built automated maintenance workflows using AWS Systems Manager and tag-based fleet discovery.
• Added monitoring, snapshots, and recovery tooling to support safer operational changes.

Reliability / Production Concerns

• Designed controlled updates with preflight checks, snapshots, and post-change verification.
• Used event-driven recovery patterns involving instance replacement, EBS, and tag invariants.
• Kept rollback paths explicit through Route53 backups and reversible scripts.

Results

Delivered a more reproducible and controlled AWS web estate, with infrastructure-managed versions, automated operational workflows, and safer change management.

Trade-offs

• Some production operations remained tied to AWS account and runtime discovery complexity.
• Recovery tooling relies on cloud-native services rather than a fully custom orchestration layer.
• Exact production telemetry and cost metrics are not claimed without direct evidence.

What I Would Change Today

• Continue improving runtime discovery contracts and fleet tagging.
• Add stronger event correlation and recovery orchestration.
• Improve secret management and deployment validation automation.
• Standardize rollback scripts with more automated verification.

Technology

AWS, Terraform, ECS, ALB, CloudFront, Route53, RDS, EBS, S3, Secrets Manager, ACM, CloudWatch, SNS, Systems Manager, Lambda, EventBridge.