omdb
patreon
Meeting with CloudFlare this week...
🕑 Added 2017-05-03 19:23:12 +0000 UTC I'm meeting with CloudFlare on Friday to discuss our options going forward using their services. Unfortunately the API is 100% dependent on their caching, even with the new server.We are going to analyze the traffic with some simulated rate limiting rules applied and calculate the costs, they already warned me that the traffic I'm currently getting would result in a "high bill", and they want to discuss a discounted flat rate option. However, I already know this discounted flat rate option will still be in $1k-3k+ price range, which this Patreon is nowhere near covering those kind of costs.
I only see two viable options moving forward, and that's what this poll is about.
Option 1: Require registration/API Keys and enable rate limiting
Option 2: Go private with no rate limiting for ALL patrons
Comments
Johan Ramael
Hi Brian, I'd love to test. If you send me the URL I'll give it a run every day.
Brian Fritz
That's an ongoing issue I'm currently investigating with two other users, it has NOTHING to do with my server or the rate limiting, in fact when the users bypass Cloudflare and talk directly to my server they are having zero timeouts etc. If you would like to test this also I would appreciate the extra data point as I'm currently gathering all of the information I can from users experiencing this before contacting Cloudflare. Thanks
Johan Ramael
Ever since the API went paying, I cannot gat any batch processing done anymore and getting worse every day. After a few dozen requests I'm not getting any response anymore on my HTTP requests and I need to wait and continue. Sorry, but moving from free to paying, OK, but giving worse service, no can do. I this rate limiting / dropping requsts continues, I'm out.
Brian Fritz
Sorry for the inconvenience, I basically had to disincentivize people from using it to quickly reduce traffic by 90%. Once the traffic gets under control, and I finish the rate limiting code I'm going to create a nice registration process to give free keys with limits.
Arthur Guy
You should be rate limiting even with Patreon. It's just bad practice to have an API wide open, on both your end and the users end. I use this API on a website for a major movie theater on a VERY busy website and hit the API less than 50 API calls a month. Oh well, guess I'm paying now.
sagar baranwal
Just signed up for paid plan still it shows wrong api
Marcel van der Veldt
As the author of the metadata module that is used in Kodi to show the OMDB info I think a hybrid option would be great. Users want to have all bells and whistles, why not pay a few bucks if they want it updated. FYI: I have implemented a 2 weeks cache in my module a while ago. My suggestion would be that users get like 5 requests free a day (or something like that) and if they want the most updated information without any limits, they must be a patreon.
Box Office Buz
Interesting ! Going to have to look into the cached response. I don't have even close to enough people using my current API that would require it yet, but I am working on V2 and I think that one will be much more popular.
Marco
Well, let Redis cache the responses of course. Redis can easily handle 10-20K requests per second on my laptop. And perhaps put Varnish in front. Varnish has modules that handle rate limiting via API keys & IP's. What I'm saying is that instead of just throwing bigger servers at it, I suggest looking at a different approach. (And I also think Rate limiting using should be avoided when possible. It requires people to update their scripting or roll out new versions of their Apps as well.)
Box Office Buz
What would the redis be used for ? Curious as I am building an API for my own website. Right now I just use redis to cache video view counts.
Douglas Alves
Sounds really great!
Josar
Good suggestion!
Josar
I think that Option 1 is a good start, and see how the traffic gets from there. I know there are Kodi addons that use your API, and I can imagine the abuse and amounts of hits that is producing. Then if that still doesn't help, Option 2 could be implemented.
Marco
For your kind of data (and especially when not using API keys), why not put Redis in front? It's easy to setup. And when properly done, a single instance can handle much more than you need right now.
Box Office Buz
Fingers crossed for sure.
Brian Fritz
I'm looking into it now, I don't know if they play nice with CloudFlare and AWS is NOT an option (Amazon's lawyers don't like me very much).
Brian Fritz
@Matt That's what I'm hoping for also. However, I think Option 1 only buys me a limited amount of time with a lot of unknown costs due to the rate limiting.
Peter Xander
Hybrid sounds good for me too!
Matt Sidor
This might have the added benefit of encouraging more donations
Box Office Buz
On a slight side note. Have you thought about something like "mashape" ? You would DNS threw them, they attach a header to ever request, and they qould actually take care of the amount of traffic one user can utilize. Than say (over 10,000 requets a day) you can charge people, and mashape actually takes care of that too.
Box Office Buz
The only thing I could think of is combining layer 7 load balancing with cloudflare.
Hans Ecke
That sounds good Matt
Matt Sidor
What about some hybrid of both? API keys with X number of free requests per day; after that you will need to donate to unlock unlimited requests.
Brian Fritz
The traffic is out of control! Weekdays it's getting over 1.5 billion requests (24h) with peaks of 100 million requests per hour. I did some tests bypassing the CDN and the servers got crushed instantly. It's truly a testament to how powerful CloudFlares services actually are.
Box Office Buz
I wish there was a way I could help. I would say I could help with setting up load balancers for the API, but if you are usizing cloudfalre I am not 100% on what that would involve.
