Paging API

Welcome Matt!

I didn't think we would need pagination in any of the API calls we currently provide, though I can understand that the episode list for such a show that has been running since forever might provide a bit of a challenge on low powered devices.

I'm not very enthusiastic yet about adding this kind of pagination though, because it would mess with our caching strategy: for performance reasons, all API requests are heavily cached, but allowing for this kind of pagination would generate an almost infinite amount of page combinations, which would cause a large amount of cache misses.

What kind of application are you currently building? Maybe we can help you come up with a different approach that doesn't require pagination on our end. For example, a lot of JSON parsers support streaming, which lets you parse a JSON file in small increments instead of loading the entire tree in memory. For example: http://wiki.fasterxml.com/JacksonStreamingApi

Hi David

I fully understand your position re caching.

For background, I am not creating an application, more exploring a framework that can represent a virtual infinite set of data efficiently in a client HTML application (the code is available here https://github.com/m5p3nc3r/backbone.cvp). I was looking for a suitable online data set to experiment with, and TV content seems like a sensible source data set due to the vast volumes of data it can represent.

There are other issues to consider when designing an API for consumption on mobile or embedded devices such as DTV or STB:

- Memory footprint, historically these tend to be cost sensitive devices, so local memory tends to be very limited.

- Network delivery - on mobile, every byte counts. Limiting the data response to that which is required for the application will improve overall application performance

- Power - the less data the application has to download and process in order to fulfill its needs, the less power the device will use, and the longer the battery will last

Now I fully understand that none of these are a primary concern of the service you are constructing, but it might be useful to have a discussion around how a paging based solution could be implemented in a CDN/caching efficient way?

A starter for 10 (cultural reference there, sorry!) How about having the server quantize to server defined cache efficient boundaries - so:

<fetch URL> ? start=55 & count=7

would return

{ header: { start: 50, count: 20, total:567}, data: [<response data array>]}

Quantizing both the start and count to the nearest 10 whilst ensuring the response data set covers the requested range?

I understand that this still creates a large number of response cache points, but is would be much more manageable?

Ah, that's an interesting use case! Though yeah, it's probably a little different from the average user of our API. Your arguments are all completely valid, but in our case I think the vast majority of the users requesting a show's episode list want the entire list anyway, not a paged subset of it.

Anyway, limiting the pagination to a predefined set of pages would be a lot more manageable, but then I would prefer a less arbitrary method of pagination, such as fetching all episodes for a given season. This is something we could implement in the future, if there's demand for it.

If anyone else is interested in a form of pagination for the episode list, do weigh in :)