Site architecture
I designed, provisioned and deployed gridviz end to end on AWS, to learn as much about AWS as possible while keeping it cheap to run.
Request flow
Browser
└─▶ Route 53 andyouandyou.com
└─▶ CloudFront HTTPS · cache
├─▶ S3 site bucket
├─▶ S3 data bucket (/data/*)
│ latest.json latest interval (Live)
│ days/<date>.json one file per trading day (Live, Plots, Solve)
│ plots/<measure>-daily daily means (Plots)
│ plots/<measure>-extremes highest and lowest days (Plots)
└─▶ Lambda solve (/api/solve?region=…&date=…)
private function URL, cached per region and day
reads days/<date>.json → Pyomo + HiGHS → schedule and bid (Solve)
EventBridge Scheduler (every 5 min)
└─▶ Lambda ingest AEMO NEMWeb → latest.json + today's day file
EventBridge Scheduler (00:30 daily)
└─▶ Lambda rebuild day files → plot filesAWS services
| Route 53 | DNS for andyouandyou.com |
|---|---|
| Certificate Manager | HTTPS certificate, renewed automatically |
| CloudFront | CDN serving the site, /data/* and /api/* |
| S3 | Two private buckets: the site, and NEM data |
| EventBridge Scheduler | Runs ingest every 5 minutes and the rebuild nightly |
| Lambda | gridviz-ingest fetches AEMO data, gridviz-rebuild makes the plot files, and gridviz-solve runs battery optimisations on demand |
| CloudWatch | Logs, an error alarm per Lambda, and an alarm if ingest stops running |
| Pyomo + HiGHS | Python modelling and open-source MIP solver behind gridviz-solve |
| Next.js static export | Frontend as plain HTML and JS, charts with Plotly |
Delivery pipeline
- Pull requests: GitHub Actions runs lint, tests, type checks and a build, plus the Python solver's tests (pytest via uv).
- Releases: a v*.*.* tag builds the site, syncs it to S3 and invalidates CloudFront.
- Infrastructure: CloudFormation and SAM, each change previewed as a change set.
- CI's IAM user can only update the site bucket and invalidate the distribution.
- The Python solver is bundled with uv for arm64 Lambda and deployed with the data stack.
Decisions and trade-offs
- Static JSON in S3 instead of a database: costs cents a month.
- Serverless instead of servers: the only fixed cost is Route 53, about A$1 a month.
- Two years backfilled from AEMO's archives, checked against AEMO's own aggregates.
- Plot files rebuilt nightly; the browser adds the days since.
- Market events are flagged from AEMO's own data fields, not a hand-made list.
- Battery solves run on demand rather than precomputed: one region-day solves in about a second, and CloudFront caches each answer.
- The solve Lambda sits behind CloudFront with origin access control on a GET-only function URL, so there's no API Gateway, no CORS, and nothing can call it directly.
- HiGHS through Pyomo instead of GAMSpy: no licence to manage in Lambda, and no model size limits.
- Reserved concurrency of 2 caps what a flood of requests to the public solve endpoint could cost.
- The account's SCPs block us-east-1 CloudFormation and OIDC, so the certificate was made by hand and CI uses a scoped IAM user.