API Request Timeout

@wezley98 wrote:
Hi,

Since the changes to the ddos setup I've noticed all the requests from my Android App (Episode Guide) are timing out. In the background I'm using OkHttp v4 with Retrofit and RxJava which automatically supports http2 and multiplexing but yet I'm still seeing errors. Even for single requests which work fine in a browser. Have any special restrictions been placed on mobile apps or okhttp client headers?

E/intercept: Request failed - 1 https://api.tvmaze.com/shows/36947/cast
D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/36947/cast
E/intercept: Request failed - 2 https://api.tvmaze.com/shows/36947/cast
D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/36947/cast
E/intercept: Request failed - 3 https://api.tvmaze.com/shows/36947/cast
D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/36947/cast
E/RESULT ERROR: error thrown: timeout

Regards

Wez

Even setting the connection pool manually, doesn't seem to help

OkHttpClient.Builder()
    .connectionPool(ConnectionPool(0,1,TimeUnit.MILLISECONDS))

https://stackoverflow.com/questions/63047533/connection-pool-okhttp

No, we don't block based on headers or anything of the kind. Are all requests timing out, or only after a certain point?

Can you share the IP address of the device where the requests time out?

@david wrote:
No, we don't block based on headers or anything of the kind. Are all requests timing out, or only after a certain point?

Can you share the IP address of the device where the requests time out?

Hi David,

Thanks for looking into this for me. I've just tested again and it seems to be working now, did you make any changes your side?

No, no changes in the past couple of weeks. If it happens again, let me know an IP address and I'll check our logs :)

I've just ran a simple test to add five shows. Here is the log output, You can see most requests pass but some are still failing.

You can see that retry 1 failed for some requests, but the fact it didn't try again must mean the second request worked.

2022-10-11 11:55:37.894 11718-11851 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/31365
2022-10-11 11:55:37.895 11718-11857 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41748
2022-10-11 11:55:37.895 11718-11845 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41748
2022-10-11 11:55:37.896 11718-11855 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41414
2022-10-11 11:55:37.896 11718-11860 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/73/episodes?specials=1
2022-10-11 11:55:37.896 11718-11859 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41749
2022-10-11 11:55:37.896 11718-11867 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41414
2022-10-11 11:55:37.896 11718-11852 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/36947/episodes?specials=1
2022-10-11 11:55:37.896 11718-11849 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/73
2022-10-11 11:55:37.896 11718-11848 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/73/episodes?specials=1
2022-10-11 11:55:37.896 11718-11856 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41748/episodes?specials=1
2022-10-11 11:55:37.897 11718-11866 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41414/episodes?specials=1
2022-10-11 11:55:37.897 11718-11853 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/36947
2022-10-11 11:55:37.897 11718-11844 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41748/episodes?specials=1
2022-10-11 11:55:37.898 11718-11847 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41749
2022-10-11 11:55:37.901 11718-11865 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/36947
2022-10-11 11:55:37.901 11718-11846 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41749/episodes?specials=1
2022-10-11 11:55:37.902 11718-11861 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/73
2022-10-11 11:55:37.902 11718-11854 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41414/episodes?specials=1
2022-10-11 11:55:37.902 11718-11858 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41749/episodes?specials=1
2022-10-11 11:55:37.903 11718-11863 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/31365
2022-10-11 11:55:37.904 11718-11864 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/36947/episodes?specials=1
2022-10-11 11:55:37.907 11718-11850 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/31365/episodes?specials=1
2022-10-11 11:55:37.909 11718-11862 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/31365/episodes?specials=1
2022-10-11 11:55:38.011 11718-11895 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41749/episodes?specials=1
2022-10-11 11:55:38.011 11718-11896 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41749
2022-10-11 11:55:38.012 11718-11893 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41748/episodes?specials=1
2022-10-11 11:55:38.030 11718-11894 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41748
2022-10-11 11:55:38.034 11718-11897 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/73/episodes?specials=1
2022-10-11 11:55:38.039 11718-11901 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/73
2022-10-11 11:55:38.045 11718-11905 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/31365
2022-10-11 11:55:38.047 11718-11904 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/31365/episodes?specials=1
2022-10-11 11:55:38.063 11718-11907 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/36947/episodes?specials=1
2022-10-11 11:55:38.065 11718-11908 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/36947
2022-10-11 11:55:38.071 11718-11910 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41414/episodes?specials=1
2022-10-11 11:55:38.076 11718-11912 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41414
2022-10-11 11:55:38.959 11718-11856 D/intercept: Served from network
2022-10-11 11:55:38.968 11718-11905 D/intercept: Served from network
2022-10-11 11:55:38.968 11718-11893 D/intercept: Served from network
2022-10-11 11:55:38.987 11718-11858 D/intercept: Served from network
2022-10-11 11:55:39.005 11718-11849 D/intercept: Served from network
2022-10-11 11:55:39.031 11718-11912 D/intercept: Served from network
2022-10-11 11:55:39.054 11718-11907 D/intercept: Served from network
2022-10-11 11:55:39.078 11718-11866 D/intercept: Served from network
2022-10-11 11:55:39.108 11718-11897 D/intercept: Served from network
2022-10-11 11:55:39.137 11718-11850 D/intercept: Served from network
2022-10-11 11:55:39.154 11718-11910 D/intercept: Served from network
2022-10-11 11:55:39.176 11718-11856 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41414/images
2022-10-11 11:55:39.176 11718-11857 D/intercept: Served from network
2022-10-11 11:55:39.186 11718-11905 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41748/images
2022-10-11 11:55:39.203 11718-11904 D/intercept: Served from network
2022-10-11 11:55:39.214 11718-11893 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/31365/images
2022-10-11 11:55:39.226 11718-11852 D/intercept: Served from network
2022-10-11 11:55:39.254 11718-11844 D/intercept: Served from network
2022-10-11 11:55:39.278 11718-11865 D/intercept: Served from network
2022-10-11 11:55:39.302 11718-11855 D/intercept: Served from network
2022-10-11 11:55:39.327 11718-11846 D/intercept: Served from cache
2022-10-11 11:55:39.328 11718-11908 D/intercept: Served from network
2022-10-11 11:55:39.333 11718-11858 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/36947/images
2022-10-11 11:55:39.350 11718-11847 D/intercept: Served from network
2022-10-11 11:55:39.351 11718-11853 D/intercept: Served from cache
2022-10-11 11:55:39.356 11718-11849 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/36947/images
2022-10-11 11:55:39.377 11718-11860 D/intercept: Served from network
2022-10-11 11:55:39.400 11718-11856 D/intercept: Served from network
2022-10-11 11:55:39.431 11718-11905 D/intercept: Served from network
2022-10-11 11:55:39.460 11718-11893 D/intercept: Served from network
2022-10-11 11:55:39.479 11718-11858 D/intercept: Served from network
2022-10-11 11:55:39.508 11718-11896 D/intercept: Served from network
2022-10-11 11:55:39.538 11718-11849 D/intercept: Served from network
2022-10-11 11:55:40.309 11718-11912 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41749/images
2022-10-11 11:55:40.359 11718-11912 D/intercept: Served from network
2022-10-11 11:55:41.529 11718-11901 E/intercept: Request failed - 1 https://api.tvmaze.com/shows/73
2022-10-11 11:55:41.530 11718-11901 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/73
2022-10-11 11:55:41.536 11718-11901 D/intercept: Served from cache
2022-10-11 11:55:41.540 11718-11907 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/73/images
2022-10-11 11:55:41.567 11718-11859 E/intercept: Request failed - 1 https://api.tvmaze.com/shows/41749
2022-10-11 11:55:41.567 11718-11859 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41749
2022-10-11 11:55:41.567 11718-11848 E/intercept: Request failed - 1 https://api.tvmaze.com/shows/73/episodes?specials=1
2022-10-11 11:55:41.568 11718-11848 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/73/episodes?specials=1
2022-10-11 11:55:41.569 11718-11845 E/intercept: Request failed - 1 https://api.tvmaze.com/shows/41748
2022-10-11 11:55:41.569 11718-11845 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41748
2022-10-11 11:55:41.573 11718-11859 D/intercept: Served from cache
2022-10-11 11:55:41.574 11718-11845 D/intercept: Served from cache
2022-10-11 11:55:41.574 11718-11864 E/intercept: Request failed - 1 https://api.tvmaze.com/shows/36947/episodes?specials=1
2022-10-11 11:55:41.575 11718-11864 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/36947/episodes?specials=1
2022-10-11 11:55:41.578 11718-11848 D/intercept: Served from cache
2022-10-11 11:55:41.580 11718-11864 D/intercept: Served from cache
2022-10-11 11:55:41.588 11718-11907 D/intercept: Served from network
2022-10-11 11:55:41.589 11718-11861 E/intercept: Request failed - 1 https://api.tvmaze.com/shows/73
2022-10-11 11:55:41.590 11718-11861 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/73
2022-10-11 11:55:41.590 11718-11895 E/intercept: Request failed - 1 https://api.tvmaze.com/shows/41749/episodes?specials=1
2022-10-11 11:55:41.590 11718-11867 E/intercept: Request failed - 1 https://api.tvmaze.com/shows/41414
2022-10-11 11:55:41.590 11718-11895 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41749/episodes?specials=1
2022-10-11 11:55:41.590 11718-11867 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41414
2022-10-11 11:55:41.592 11718-11894 E/intercept: Request failed - 1 https://api.tvmaze.com/shows/41748
2022-10-11 11:55:41.592 11718-11894 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41748
2022-10-11 11:55:41.594 11718-11862 E/intercept: Request failed - 1 https://api.tvmaze.com/shows/31365/episodes?specials=1
2022-10-11 11:55:41.594 11718-11862 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/31365/episodes?specials=1
2022-10-11 11:55:41.597 11718-11894 D/intercept: Served from cache
2022-10-11 11:55:41.598 11718-11851 E/intercept: Request failed - 1 https://api.tvmaze.com/shows/31365
2022-10-11 11:55:41.598 11718-11851 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/31365
2022-10-11 11:55:41.599 11718-11862 D/intercept: Served from cache
2022-10-11 11:55:41.600 11718-11867 D/intercept: Served from cache
2022-10-11 11:55:41.603 11718-11851 D/intercept: Served from cache
2022-10-11 11:55:41.603 11718-11861 D/intercept: Served from cache
2022-10-11 11:55:41.604 11718-11895 D/intercept: Served from cache
2022-10-11 11:55:42.065 11718-11866 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41748/images
2022-10-11 11:55:42.068 11718-11866 D/intercept: Served from cache
2022-10-11 11:55:42.109 11718-11850 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41749/images
2022-10-11 11:55:42.113 11718-11850 D/intercept: Served from cache
2022-10-11 11:55:42.154 11718-11897 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/36947/images
2022-10-11 11:55:42.158 11718-11897 D/intercept: Served from cache
2022-10-11 11:55:43.079 11718-11854 D/intercept: Served from network
2022-10-11 11:55:43.268 11718-11863 D/intercept: Served from network
2022-10-11 11:55:44.975 11718-11857 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/73/images
2022-10-11 11:55:44.980 11718-11904 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41748/images
2022-10-11 11:55:44.987 11718-11857 D/intercept: Served from cache
2022-10-11 11:55:44.989 11718-11904 D/intercept: Served from cache
2022-10-11 11:55:45.006 11718-11852 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41414/images
2022-10-11 11:55:45.016 11718-11852 D/intercept: Served from cache
2022-10-11 11:55:45.155 11718-11844 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/31365/images
2022-10-11 11:55:45.159 11718-11844 D/intercept: Served from cache
2022-10-11 11:55:45.429 11718-11865 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/73/images
2022-10-11 11:55:45.434 11718-11865 D/intercept: Served from cache
2022-10-11 11:55:45.438 11718-11855 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41749/images
2022-10-11 11:55:45.446 11718-11855 D/intercept: Served from cache
2022-10-11 11:55:45.479 11718-11846 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/41414/images
2022-10-11 11:55:45.488 11718-11846 D/intercept: Served from cache
2022-10-11 11:55:45.567 11718-11908 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/31365/images
2022-10-11 11:55:45.580 11718-11908 D/intercept: Served from cache
2022-10-11 11:55:51.666 11718-11853 D/intercept: TVmaze - Setting 1 day cache time, not premium request: https://api.tvmaze.com/shows/73/cast
2022-10-11 11:55:51.717 11718-11853 D/intercept: Served from network

If all of the initial requests succeed, but only at some point requests start to intermittently fail, the odds are almost 100% that you are not actually properly re-using or multiplexing the HTTP connections.

I see every request is ran in a different thread. Are you sure these various threads are even re-using the same okhttp instance?

Regardless, it seems you're firing off 36 requests to our API within 200 milliseconds. Please refer to our guide on rate limits (https://www.tvmaze.com/api#rate-limiting): 

API calls are rate limited to allow at least 20 calls every 10 seconds per IP address. If you exceed this rate, you might receive an HTTP 429 error. We say at least, because rate limiting takes place on the backend but not on the edge cache.

Under special circumstances we may temporarily have to impose a stricter rate limit. So even if your client wouldn't normally exceed our rate limit, it's useful to gracefully handle HTTP 429 responses: simply retry the request after a small pause instead of treating it as a permanent failure.

I'd advise you to either fire these requests serially in a single thread, or if you want to run them in parallel (at the risk of exceeding our rate limits, so be sure to have a proper back-off and retry mechanism) use something like a queue with a capped amount of workers.

Hi David,

Thanks for the useful feedback, I've switched the sync job to use Scheduler.single() (Returns a default, shared, single-thread-backed Scheduler instance for work requiring strongly-sequential execution on the same background thread.)

http://reactivex.io/RxJava/3.x/javadoc/io/reactivex/rxjava3/schedulers/Schedulers.html#single--

So hopeful this will help. I'll finish some testing and roll it out to production soon.