Skip to content

Site architecture

I designed, provisioned and deployed gridviz end to end on AWS, to learn as much about AWS as possible while keeping it cheap to run.

Request flow

Browser
  └─▶ Route 53         andyouandyou.com
       └─▶ CloudFront  HTTPS · cache
            ├─▶ S3     site bucket
            ├─▶ S3     data bucket (/data/*)
            │            latest.json                latest interval (Live)
            │            days/<date>.json           one file per trading day (Live, Plots, Solve)
            │            plots/<measure>-daily      daily means (Plots)
            │            plots/<measure>-extremes   highest and lowest days (Plots)
            └─▶ Lambda solve (/api/solve?region=…&date=…)
                         private function URL, cached per region and day
                         reads days/<date>.json → Pyomo + HiGHS → schedule and bid (Solve)

EventBridge Scheduler (every 5 min)
  └─▶ Lambda ingest    AEMO NEMWeb → latest.json + today's day file

EventBridge Scheduler (00:30 daily)
  └─▶ Lambda rebuild   day files → plot files

AWS services

Route 53DNS for andyouandyou.com
Certificate ManagerHTTPS certificate, renewed automatically
CloudFrontCDN serving the site, /data/* and /api/*
S3Two private buckets: the site, and NEM data
EventBridge SchedulerRuns ingest every 5 minutes and the rebuild nightly
Lambdagridviz-ingest fetches AEMO data, gridviz-rebuild makes the plot files, and gridviz-solve runs battery optimisations on demand
CloudWatchLogs, an error alarm per Lambda, and an alarm if ingest stops running
Pyomo + HiGHSPython modelling and open-source MIP solver behind gridviz-solve
Next.js static exportFrontend as plain HTML and JS, charts with Plotly

Delivery pipeline

  • Pull requests: GitHub Actions runs lint, tests, type checks and a build, plus the Python solver's tests (pytest via uv).
  • Releases: a v*.*.* tag builds the site, syncs it to S3 and invalidates CloudFront.
  • Infrastructure: CloudFormation and SAM, each change previewed as a change set.
  • CI's IAM user can only update the site bucket and invalidate the distribution.
  • The Python solver is bundled with uv for arm64 Lambda and deployed with the data stack.

Decisions and trade-offs

  • Static JSON in S3 instead of a database: costs cents a month.
  • Serverless instead of servers: the only fixed cost is Route 53, about A$1 a month.
  • Two years backfilled from AEMO's archives, checked against AEMO's own aggregates.
  • Plot files rebuilt nightly; the browser adds the days since.
  • Market events are flagged from AEMO's own data fields, not a hand-made list.
  • Battery solves run on demand rather than precomputed: one region-day solves in about a second, and CloudFront caches each answer.
  • The solve Lambda sits behind CloudFront with origin access control on a GET-only function URL, so there's no API Gateway, no CORS, and nothing can call it directly.
  • HiGHS through Pyomo instead of GAMSpy: no licence to manage in Lambda, and no model size limits.
  • Reserved concurrency of 2 caps what a flood of requests to the public solve endpoint could cost.
  • The account's SCPs block us-east-1 CloudFormation and OIDC, so the certificate was made by hand and CI uses a scoped IAM user.