sondehub
patreon
More affordable backend
🕑 Added 2021-05-28 12:32:27 +0000 UTCIn the release retrospect I talked a bit about reducing cost. As it stands Patreon doesn't cover the cost to run SondeHub and I make up the difference from my own pocket, so there is incentive from me to ensure whatever is implemented is not just maintainable but also affordable.
I've implemented the changes previously mentioned, which included removing AWS IoT actions from the ingestion pipeline, and batching up SQS messages. It's looking like these changes have saved roughly a third from bill!
Batching up SQS messages isn't without its own problems though. Since the queue worker make encounter an error processing a single payload it has to decide:
- fail the entire batch of payloads
- add a new message to the queue with the failed messages
- make that batch as successful
The backend code isn't very robust at the moment for this, and we recently saw a message get stuck. In this case this was a station running an older version of radiosonde autorx and falsely decoded the datetime field for a DFM sonde.
Nothing bad happened in this case, it just meant that we kept trying to process it.
We also have a similar issue when it comes to accepting payloads on our HTTP endpoint. As we accept many payloads in one request, we need to return something to the client to give it an idea of what payloads successfully were uploaded, and hopefully provide more useful error messages. At the moment we just send back a 200 OK or a 503 on any failure. Likely need to fix in sooner rather than later.
Terraform
I've also taken the time to get real world and terraform a little more aligned. There's still a handful of resources that need to be terraformed (and also move some resources from my personal AWS account to the dedicated AWS account, such as the `/card` lambda function, and main CloudFront distribution)
Comments
Chandler Heath
How much is the funding gap that needs to be closed?
