Designing a Resilient Third-Party Data Ingestion Platform

Built and led a backend-first analytics platform that normalized and persisted third-party music and social metrics, reducing request-time dependency on rate-limited provider APIs while supporting historical analytics and a Next.js product experience.

My Role

Technical Lead — architecture, backend engineering, integrations, and production delivery.

Problem

External music and social APIs exposed inconsistent schemas, rate limits, latency, and intermittent failures, while the business needed reliable historical analytics and a product layer for artist management.

Context / Constraints

• Provider APIs had quota limits, 429 responses, and variable rate-limit behavior.
• Historical analytics required persisted snapshots rather than live-only requests.
• The platform needed account administration, artist onboarding, and calendar-based workflows.
• Delivery had to remain practical for a small team and existing AWS EC2 infrastructure.

Architecture

The solution separated the portal and analytics boundaries:

• Portal: Next.js + React with NextAuth, MongoDB/Mongoose for account and content state, and AWS S3/SES for assets and notifications.
• Analytics backend: Node.js + Express, PostgreSQL + Sequelize for normalized metric storage, scheduled ingestion jobs, and analytics endpoints.

Key Engineering Decisions

• Moved core analytics toward scheduled, persisted metric snapshots, reducing request-time dependency on external providers.
• Implemented provider integrations with rate-limit-aware retry/backoff for Chartmetric and request pacing for Instagram/Meta Graph API.
• Used PostgreSQL for normalized historical analytics and MongoDB for portal/account state to preserve clearly separated persistence responsibilities.

Implementation

• Built a Node.js/Express backend to manage provider connections, account data, artist metadata, and ingestion workflows.
• Implemented scheduled ingestion jobs that normalized provider data into a common historical schema.
• Added platform abstractions for accounts, artists, platforms, metrics, content, content metadata, and refresh tokens.
• Built a Next.js/React frontend for authentication, onboarding, analytics dashboards, and calendar-based planning.

Reliability / Production Concerns

• Implemented explicit timeout and retry behavior around provider HTTP requests.
• Added request pacing and 429/504 handling for Chartmetric integrations.
• Designed indexed historical storage to support date-window queries and stable analytics retrieval.

Results

Delivered a backend-first analytics ingestion platform with persisted snapshots for historical metrics, reduced runtime dependency on provider APIs, and a product-ready Next.js experience.

Trade-offs

• Scheduled jobs execute in-process rather than on a separate durable queue.
• Provider abstraction could be stronger to reduce duplicated retry logic.
• Observability and durable retry ledger support remain limited in the current implementation.

What I Would Change Today

• Introduce a durable job queue such as SQS or BullMQ.
• Separate ingestion workers from the web process.
• Add durable retry records, DLQ handling, and stronger idempotency guarantees.
• Improve structured logging and OpenTelemetry tracing.

Technology

Node.js, Express, PostgreSQL, Sequelize, REST APIs, third-party integrations, scheduled jobs, PostgreSQL indexing, Next.js, React, MongoDB, Mongoose, AWS S3, AWS SES, GitHub Actions, EC2, PM2, Nginx.