sondehub
patreon
Station icon updates and ingestion filtering
🕑 Added 2021-08-20 04:47:12 +0000 UTCStation Icons
I haven't been feeling too well lately so not a huge update today but I did want to provide an update with a few things that have changed recently.
As you would have seen we ended up using the circles for the station icons. There's multiple reasons for this and we really valued your feedback. We tried a few concepts with the icon but the biggest problem we ran into is how it looked when scaled down, or on low resolution displays. Along with this, displaying the station icon actually takes a bit of a performance hit compared to just the plain circle. Once again many thanks to Luke for getting this all sorted on the frontend
With the icon problem solved the next part was colouring the circles based on patreon status. We had to build a small api that would match up reward statuses. We use the patreon API to check the note field for each user to determine which icon can be displayed. We have some flexibility here in that we could bring back pictures for certain patreon reward levels, but we haven't explored this further.
Ingestion Filter
When we built the V2 API for SondeHub we focused on getting the bare minimum features to switch over to our own backend. One of the things that was left off was implementing payload filtering which means we've ended up with some bad data coming into our system. Until now we've be relying on the client software to correctly filter out the data - this doesn't work if users change or remove the checks however.
There's multiple reasons we might get bad data:
- Incorrectly decoded frames - poor radio reception, interference, protocols with not enough error detection
- GPS noise / jamming - when the radiosonde reports back the incorrect position due to faulty GPS readings
- Multiple radiosonde on the same frequency - DFMs on the same frequency don't include their serial number in every frame so the data flip flops between serial numbers
So we now have some basic checks that occur when you upload data:
- Lat/Lon aren't 0,0 (null island)
- Altitude isn't over 50km (a weather balloon should never reach this high. This is SondeHub not RocketHub)
- Altitude isn't exactly 0m (it's very rare that a radiosonde would be at exactly 0)
- Satellite count < 4 (if we don't have a good fix then we discard it)
- The date and time is correct, within a few hours
- The serial number is correct format for the type
- Outlier detection using z-score
With all these checks we sometimes throw out legitimate data, but it should be so small compared vast amount of data we do capture that it shouldn't impact anything. The benefits are huge though and we should see much higher quality of data entering SondeHub.
