Real-time sentiment monitoring for incoming messages, built serverless on AWS. Messages land in S3, a Lambda scores them with Amazon Comprehend, the result is routed by sentiment through SQS and persisted to DynamoDB, and a Next.js console visualizes the stream live.
A portfolio project for cloud / DevOps: event-driven architecture, IaC, least- privilege IAM, a dead-letter queue, observability, and a gated CI/CD pipeline, designed to run inside the AWS Free Tier.
Status: example project, not actively deployed. This stack was built, deployed end-to-end to AWS, recorded, and then torn down with
terraform destroyso it doesn't keep costing money. There is no live URL. The video below shows it running for real. Everything needed to bring it back up is in this repo (see Deploy).
▶ Watch the deployment walkthrough (Google Drive)
The recording shows the deployed stack working end to end: messages seeded into the S3 inbox, scored by the processor Lambda, and showing up live on the CloudFront-hosted dashboard.
ObjectCreated
┌─────────┐ event ┌──────────────┐ poll ┌───────────────────────┐
│ S3 │─────────▶│ SQS ingest │──────────▶│ Lambda: processor │
│ inbox │ │ (+ DLQ) │ (batch) │ Comprehend sentiment │
└─────────┘ └──────────────┘ └───────────┬───────────┘
│
route by sentiment ┌──────────┴──────────┐
▼ ▼
┌────────────────────┐ ┌──────────┐
│ SQS route-positive │ │ DynamoDB │
│ / negative / │ │ results │
│ neutral / mixed │ └────┬─────┘
└────────────────────┘ │ Query (GSI)
▼
┌───────────────────┐ GET /messages ┌────────────────────────────┐
│ CloudFront + S3 │◀──────────────────│ API Gateway + read Lambda │
│ (Next.js console) │ (poll every 4s) └────────────────────────────┘
└───────────────────┘
Why S3 → SQS → Lambda (instead of S3 triggering Lambda directly): the queue gives you a retry buffer, batching, backpressure, and a dead-letter queue for poison messages. Direct invocation gives you none of that.
- Ingest. A text file is written to the private inbox S3 bucket (the seeder
script does this for demos). S3 fires an
ObjectCreatednotification to the ingest SQS queue. A queue resource policy only accepts messages from that exact bucket in that exact account, which blocks confused-deputy attacks. - Buffer and retry. The ingest queue has a 180 s visibility timeout (well above the Lambda's 30 s timeout) and a redrive policy: after 5 failed receives a message moves to the dead-letter queue.
- Process. An event source mapping hands the processor Lambda batches of up
to 10 messages (5 s batching window). For each record it:
- unwraps the S3 event from the SQS body and reads up to 5 KB of the object,
- calls
comprehend:DetectSentimentto get a label (POSITIVE,NEGATIVE,NEUTRAL,MIXED) and per-class scores, - writes the record to DynamoDB with
attribute_not_exists(id), so redelivered messages are detected and skipped (idempotent writes), - forwards the record to the matching
route-<sentiment>SQS queue, where downstream consumers (alerts, support tooling, etc.) could subscribe.
- Partial batch failure. The handler returns
batchItemFailures, so one bad record goes back to the queue on its own and the rest of the batch still succeeds. - Read. The dashboard calls
GET /messageson an API Gateway HTTP API. The read Lambda queries theby-recencyGSI newest-first (ScanIndexForward=False, default 60 items, capped at 200), converts DynamoDBDecimals to floats and returns JSON with CORS headers. - Visualize. The Next.js console, a static export served from a private S3 bucket through CloudFront, polls the API every 4 seconds and renders the live feed, per-sentiment counts and filters.
| Layer | AWS service | Where | Notes |
|---|---|---|---|
| Inbox storage | S3 | infra/s3.tf | Public access fully blocked; ObjectCreated:* → ingest queue |
| Queueing | SQS (ingest + DLQ + 4 routing queues) | infra/sqs.tf | Redrive after 5 receives; routing queues created with for_each |
| Sentiment scoring | Lambda (Python 3.12) + Comprehend | handlers/processor/, infra/lambda.tf | 256 MB / 30 s; partial batch failure reporting |
| Results store | DynamoDB (on-demand) | infra/dynamodb.tf | PK id = {s3_key}#{version_id}; GSI by-recency on pk + receivedAt |
| Read API | API Gateway HTTP API + Lambda | handlers/read_api/, infra/apigateway.tf | HTTP API (v2) for lower cost than REST; access logs to CloudWatch |
| Dashboard | Next.js 15 static export on S3 + CloudFront | apps/web/, infra/cloudfront.tf | Origin Access Control (SigV4) scoped to the one distribution; HTTPS redirect |
| Permissions | IAM | infra/iam.tf | One role per Lambda, each action scoped to specific ARNs |
| Observability | CloudWatch + SNS | infra/cloudwatch.tf | 30-day log retention, DLQ-depth alarm → email, overview dashboard |
| CI/CD | GitHub Actions + OIDC | .github/workflows/ | No long-lived AWS keys; deploy is manual and confirmed |
| Demo driver | Python + boto3 (uv) | tooling/ | Uploads sample messages to the inbox bucket |
apps/web/lib/types.ts defines the record shape. The processor writes it, the read API returns it, and the dashboard renders it:
interface MessageRecord {
id: string; // "{s3_key}#{version_id}"
snippet: string; // first 280 chars of the message
sentiment: "POSITIVE" | "NEUTRAL" | "NEGATIVE" | "MIXED";
confidence: number; // Comprehend score for the winning label, 0–1
source: string; // S3 object key
receivedAt: string; // ISO 8601 S3 event time
}DynamoDB stores scores as Decimal. Both Lambdas share a decimals_to_float
helper so JSON serialization works on the way out.
- Least-privilege IAM. The processor can only
GetObjectin the inbox bucket, consume the ingest queue,SendMessageto the four routing queues,PutItemon the results table and write to its own log group. The read API only has read access (Query,Scan,GetItem) to the results table. The single"*"resource iscomprehend:DetectSentiment, because Comprehend has no resource-level permissions. - No public buckets. Both buckets block all public access. CloudFront reads the frontend bucket through OAC, conditioned on the distribution ARN.
- Scoped invoke permissions. API Gateway can invoke the read Lambda only from this API's execution ARN.
- No secrets in the repo.
backend.tf,*.tfvarsand.envfiles are gitignored. CI authenticates to AWS with GitHub OIDC and writes backend and variable files from repository secrets at runtime.
- Structured, prefixed log lines (
[PROCESSING],[COMPREHEND OK],[DUPLICATE],[ROUTED],[FAILED]…) make a single message easy to trace in CloudWatch Logs. - A CloudWatch alarm fires by email (SNS) if anything reaches the dead-letter queue.
- A CloudWatch dashboard tracks Lambda invocations, errors, throttles and p50/p99 duration, plus ingest queue throughput, backlog and DLQ depth.
ci.ymlruns on every push and PR: frontend lint, type-check, Vitest unit tests and static build;terraform fmt -checkandvalidate;ruffand a compile check on the Python handlers.deploy.ymlruns only on manual dispatch, and you have to typedeployto confirm. It assumes an AWS role over OIDC, runsterraform planandapply, builds the dashboard with the live API URL, syncs it to S3 with separate cache headers for hashed assets and for HTML/JSON, and invalidates CloudFront.
Operational details (OIDC setup, secrets, rollback, teardown) are in runbook.md.
apps/web/ Next.js dashboard (static export → S3 + CloudFront)
handlers/ Lambda source (Python): processor + read_api
infra/ Terraform: one file per concern (s3, sqs, lambda, iam, ...)
tooling/ Python seeder + sample messages (uv)
.github/ CI and gated deploy workflows
runbook.md Deploy, verify, rollback and teardown procedures
Who built what: I wrote the Lambda handlers, the Terraform and the CI/CD pipelines. The dashboard UI was AI-generated, and the seeder was a joint effort.
cd apps/web
npm install
npm run dev # http://localhost:3000With no NEXT_PUBLIC_API_URL set, the dashboard runs on built-in mock data, so
you can explore the UI without a backend. Since the project is no longer
deployed, this is the easiest way to see it.
- Node 20+ (
apps/web/.nvmrc) - Terraform ≥ 1.7
- AWS CLI, configured with credentials (
aws configure) - Python 3.12 +
uv(for the seeder) - An AWS account (everything here targets the Free Tier)
To redeploy it yourself, either run the Deploy GitHub Actions workflow (see
runbook.md) or do it by hand:
# 1. State backend (one-time): create an S3 state bucket + DynamoDB lock table,
# then: cd infra && cp backend.tf.example backend.tf (fill in names)
# 2. Vars
cp infra/terraform.tfvars.example infra/terraform.tfvars # set unique bucket names + alarm email
# 3. Build the infra
cd infra
terraform init
terraform plan
terraform apply
# 4. Ship the dashboard
cd ../apps/web
echo "NEXT_PUBLIC_API_URL=<api_endpoint from terraform output>" > .env.local
npm install && npm run build # produces out/
aws s3 sync out/ s3://<frontend_bucket> --delete
aws cloudfront create-invalidation --distribution-id <id> --paths '/*'
# 5. Drive it
cd ../../tooling
uv sync
uv run python src/seed_messages.py --bucket <inbox_bucket> --count 12Designed for ~$0 idle under the Free Tier. There is no VPC, NAT gateway, ALB, RDS, or Fargate. Those are the services that bill around the clock, and this project leaves them out on purpose. Caveats:
- Comprehend's 5M-character free tier is first-12-months only; after that it's ~$0.0001/unit. Pennies for a demo, but not literally zero forever.
- The real risk on a "free" project is leaving it running, which is why this one has been torn down now that it has been demonstrated.
aws s3 rm s3://<inbox_bucket> --recursive
aws s3 rm s3://<frontend_bucket> --recursive
cd infra && terraform destroyThen confirm in the console that the S3 buckets, DynamoDB table, Lambdas, queues, and CloudFront distribution are gone, and check Billing the next day.