Videochat Extension
patreon
ome.tv attack (?) on Videochat Extension - status report for 2024-07 (July 2024)
🕑 Added 2024-07-27 17:58:55 +0000 UTCMost of the information about the incident is already here, so I'll do my best not to repeat myself and save your time.
What happened?
An anonymous script flooded the channel used to transmit potential IP addresses for webrtc connections from the page to the extension with thousands of random IP addresses per each real one, resulting in thousands of geolocation requests and possible lags on the client side.
Overall, over a billion useless requests were made among the whole userbase, but the infrastructure (mostly Cloudflare) could handle it without issues. However, there could be lagging on the client side because the extension code was not optimized for your interlocutor to have thousands of possible locations.
It was not active at all times, so we linked it to the possible human mod team work schedule.
It was loaded from the ome.tv's page side, but no idea what made it load exactly. We did not care much to investigate. We thought of possible XSS-like attack from the interlocutor initially causing issues, but the goal was to focus on hardening the defense as fast as possible.
Ome.tv could be full of vulnerabilities. When we found a simple one and tried to report it in various ways, there was no response, and it is still there. A year or two have passed since then. So, it's no wonder that such things could happen. For example, in the past, we have encountered interlocutor with the ability to disable the report button on your side.
But in this case, it is direct malicious behavior coming from the ome.tv side. We just don't know if we were the target because it was not active at all times. There are lots of forks using the same method for IP grabbing, and there were far more creative ways to mess with the extension since we basically coexist with the page.
---
We are the browser extension for your browser, and you installed us to work in your interest. We don't care much about what ome.tv does in its own script; it does not concern us since we intend to work only with browser elements, buttons, internal JavaScript APIs, and so on.
Why could that happen?
The extension needs to inject code into the page that allows the extension to detect when a new potential IP for WebRTC connection appears to process it. So, the event that was used to transmit such data to the extension was hijacked and flooded with thousands of requests. It's that simple.
We've been using this approach for over 3 years and never had any issues. We never thought about the possibility of unfriendly behavior coming from ome.tv itself.
It is very old code from a time when there were no plans for a public extension at all, so shit happens. It can happen again, but it won't cause as much havoc as it did this time.
There is currently no evidence to suggest this was a targeted attack on our project - no one contacted us, there were no prior indications, and the temporary nature of the issue (normal for hours, then an attack for hours) indicates something unusual is happening.
If it were a deliberate attack, it could have been more sophisticated. Why is this problem only prevalent at specific times? Yes, users experienced lagging sometimes, but not many reached out / cared at all.
What did we do to fix it?
Key changes include the extension using a new communication channel via the extension ID (to differentiate it from other projects) and no longer displaying new connections during an active conversation - new networks for the interlocutor, moderators, etc. Geolocation may be unstable due to this method not undergoing three years of testing with hundreds of thousands of users, but it's much better than having to process tens of millions of requests per hour on our backend.
Debug information output to the console has been reduced (no geolocation traces in the console), and many defensive checks against overload have been added.
Why do we think that it was connected to mods?
Because the attacker's script that generated random IPs had a fatal defect - not all the IPs it produced were real. Not all the possible IPs on the world are assigned somewhere.
Attacker code wanted to bypass caching, but did not care if the IPs were real or not.
So it was very easy to see when it was active and when it was not.
If the goal was to harm the extension, then why enable this thing only from time to time? We tried to guess what's going on.
The only thing we could think of is that the extension could be showing when moderator connects to our users.
We removed showing new locations during the active conversation, and will remove new locations on stop in the next release in an attempt to avoid such issues in the future.
---
It is also possible that it was intended to work at all times but breaks from time to time for some reason. However, as I said before, we did not go deep into our investigation - speed was the priority.
How much money we've lost?
We've lost $0. In fact, during the attack we welcomed 5 new patrons which is basically a 50% increase! Stonks.
You can thank the Cloudflare for that and yourself for becoming our patrons to give us strength to push this project forward.
The IP geolocation service we pay for to use for free users requests is unlimited and the very expensive patron-only geolocation service was not affected at all because of multiple additional protections.
Backup free geolocation services (low data quality / http only) were built to withstand hundreds of thousands requests per second. The most resilient backup service with low data quality does not have any rate limiting at all and was built for fun, to see how much requests per second it can process.
Still, we will have to change the behavior so that free backup services activate ONLY if our own infrastructure is down. Because no one wants billions of unintended requests even if the way to send them is no longer available.
---
We were hiding behind cloudflare and nginx, our own geolocation server could handle hundreds of requests per second, but now we can handle thousands without any issue.
We optimized our server infrastracture back in October 2023 and moved to ~20x more powerful server on July 13th.
Why the issue took so long to fix?
We reacted within a few hours originally, but thought that it was a usual DDOS attack.
We already rented a very powerful server for the patron-badge feature about a month ago, but not yet moved our geolocation service there.
So instead of thinking what could be the cause, we moved the server to 20x more powerful than before, optimized it for the high load and forgot about it. We checked if the geolocation api is working, but never actually the chat.
DDOS alerts were lost during rapid server transfer.
It was only on July 23rd when we spotted the real issue because the user actually reported it with a video.
Some other users reported the issue during the attack, but the descriptions were mixed and confusing, and I was busy.
How many users uninstalled the extension?
According to the Chrome Web Store, none.
Our uninstalls grew 23.76% compared to the previous period, and installs grew 23.77% compared to the previous period. It is the real data. So no userbase were lost!
Jokes aside, we have experienced 2000-2500 unusual uninstallations (above norm). But spike in installs compensated it, so its possible that people just reinstalled the extension again after some time.
I think we lost 0.3% - 0.5% of the user base maximum over the last two weaks. Still hurts. Sorry...
What's next?
Time will tell. No idea.
We will support you as long as you support us.
Big thanks!
---
qrluke
