confused with how to name specials (tv maze kodi scraper)

Good question! I was actually discussing this with someone on Reddit the other day as well.

To summarize: since the whole point of a special is that it doesn't have an episode number, I'm not a fan of our API clients using arbitrary numbering schemes for episode matching, since this wouldn't be very reliable.

Instead, it would probably be best to match them on airdate. That means you'd simply name the Pandemic Special "2020-09-30".

But I believe the Kodi scraper doesn't actually support this kind of matching yet though. Could you perhaps try this out and report back? Then I'll nudge the developer of the scraper here to weigh in as well and hopefully get this supported soon! :)

Thanks a lot for your quick response. Using airdate does seem to work with the scraper, so problem solved! :D

Will the kodi scraper also support the significant special changes in the future or will it be a website only feature? 

@beerland wrote:
Thanks a lot for your quick response. Using airdate does seem to work with the scraper, so problem solved! :D

Will the kodi scraper also support the significant special changes in the future or will it be a website only feature? 

Hey, that's good to hear! :)

What do you mean with "support significant special changes" exactly, though? What would you want to see differently from the way it already works right now?

Maybe I just misunderstood what actually changed though. On the website the Pandemic Special now shows as part of season 24 but on kodi its still in the normal special folder. Not a big deal though, was just curious because I thought specials would now appear in the season you were supposed to watch it.

Right, so the problem is that the Kodi scraper doesn't place significant specials in the season together with the regular episodes. I'll get that looked into!

Yep. I just tested it with another show, the Battlestar Galactica Miniseries in season 1 and the Razor special in season 3 but same thing happens.

Thanks a lot for your help, really appreciate it!

Copied from the reddit thread.  

Well that's a nice surprise, I can only theorise that the filename airdate will be matched against the relevant airdate in the xml for episode matching. That'll work for 99% of specials, only failing if there's no airdate listed or if there are two specials on one day - the very definition of an edge scenario.

I'd like some official verification or additional confirmations before promoting this because 'trust but verify' is important, but this is good news.  

The only show I know of that has released multiple minor specials on one day is Taskmaster which for several seasons released all contestant interviews on their YouTube Taskmaster channel simultaneously.  

This info should definitely be added to the kodi wiki  

https://kodi.wiki/view/Add-on:TVmaze  

Specifically because it differs from the instructions in the main kodis how to name tv shows wiki page specials section (i.e. use s00eXX). Maybe also edit this page to mention how tvmaze handles specials if this is the preferred way going forward. I still think the tvmaze web page should add a number against specials (or include it within the specials own episode page) to help those still using s00eXX format or dealing with an edge scenario, but I understand that is a design decision. I'm just happy naming by date actually works, I also feel I should have known this from reading it somewhere.  

Definitely agree! I already nudged Roman (the author of the scraper) about how we can formally document this as the way to go.

@david wrote:
Definitely agree! I already nudged Roman (the author of the scraper) about how we can formally document this as the way to go.

I believe I've thought of a major issue with using dates for specials; if a special airs the same day as a regular episode, because that occurs relatively frequently with specials. I've mulled it over while cooking my evening meal and I can't think of a way around it, which is a pain.

Supernatural finally aired it's series/season finale a day ago at 2100, at 2000 there was a minor special The Long Road Home. With just the date used in the specials filename there will be issues because the regex kodi uses just looks for date not datetime so depending on the xml episode order and whether kodi uses a first match or all occurances match a file named Supernatural.2020.11.19.The.Long.Road.Home will either be scraped as Episode s15e20 or the special or both. And I reckon the least likely match is only as the special since episodes are listed before specials.  

This is 100% a kodi issue given the way their regex looks only at date, although I wouldn't be surprised they omitted time on purpose to avoid having to deal with timezones or missing times in filenames. Either way it looks like there isn't a perfect way to deal with specials, and I know from a personal mistake that if you put both a special episode number and an airdate in a filename you end up with two copies in kodis library. 

To give context, proper handling of specials (even before Significant Specials) is an outstanding issue on the github;

https://github.com/romanvm/kodi.tvmaze.scrobbler/issues/16

@augur wrote:
I believe I've thought of a major issue with using dates for specials; if a special airs the same day as a regular episode, because that occurs relatively frequently with specials. I've mulled it over while cooking my evening meal and I can't think of a way around it, which is a pain.

That's a good point. If Kodi is aware that an episode is a special, it's easy to deal with.. just look up which episode aired on that date with type "significant_special" or "insignificant_special". I see two options, I don't know how easy they are to implement on Kodi's side...

1. Automatically detect that an episode is likely to be a special. If a directory is full of SxxExx files and one or two episodes are named 2010-01-01, those are probably specials.

2. Just tell Kodi by labeling the episode a special explicitly, "2010-01-01.Special"

More likely the kodi devs will tell you it's good enough for everyone else and we don't want to risk breaking other tv scrapers.

1 won't work, too much overhead, querying a folder, storing file list, running logic, making wooly decisions.

2 should. Since the regex is run in a specific order adding 2010-01-01.Special with a higher priority (make it the new 3) before 2010-01-01 (shift this and subsequent by +1) would work and I can't imagine it could break any other scrapers. Now getting kodi devs to make that change... rather you than me although it does have a certain logic that might appeal.

https://kodi.wiki/view/Naming_video_files/TV_shows#Single_Episodes

However there's still the multi specials on one day issue so since you're shooting for the moon how about 2010-01-01-0000.Special where 0000 is airtime in 24hr format. It could deal with multi specials released simultaneously (e.g. on YouTube) by using 0001, 0002, etc. The question is how would that work on the tvmaze website/db/api and does the kodi xml parser even accept datetime or can they only be separate.  

The problem is moreso that the current TVmaze endpoint being used only supports either:

a) season & episode numbers

b) airdate

It's not possible to use (a) because it doesn't have an episode number, therefore in the current version it just uses (b). 

When this came up before, there was discussion to allow the scraper add-on to use another endpoint if the TVmaze episode ID is known (which is set by via the metadata add-on). That way, all of the logic of identifying it is kept in that add-on and the scraper just processes it.

I misread; the thread is talking about the scraper, not the scrobbling of specials

@augur wrote:

2 should. Since the regex is run in a specific order adding 2010-01-01.Special with a higher priority (make it the new 3) before 2010-01-01 (shift this and subsequent by +1) would work and I can't imagine it could break any other scrapers. Now getting kodi devs to make that change... rather you than me although it does have a certain logic that might appeal.

A naming convention of "2010-01-01.Special" sounds good to me. Let's see what the Kodi people say.

As for multiple specials on the same day, I think those are so rare that we don't need to concern ourselves with it. Per our policy, 

Specials should be rare; if a season has nearly as many specials as regular episodes, you're most likely doing it wrong. Specials should be standalone and not in a serial format with other specials. For example, if each episode in a season is followed by an "after show" or "webisode", those episodes should be in a separate show entry.

Fair enough.  

I wrote something else (then deleted it for this) which caused me to wonder if having kodi side regex strings of 2010-01-01.Special and 2010-01-01 would both match e.g. Supernatural.2020.11.19.Special resulting in it being added to the library twice? I fear it might since, as I mentioned earlier, I once managed to add a file twice by having both episode number and airdate in the filename, could that be coded around on the kodi side or the addon?

It feels like I'm stumbling in the dark towards a solution but every step I find a coffee table waiting in ambush.

The main problem with using date pattern for specials is that a special may be released on the same day as a regular episode. In that case a pattern Foo.YYYY-MM-DD.ext will be matched to a regular episode, meaning you may end up with duplicate items in you medialibrary. That is Foo.s01e01.mp4 (released on 2020-01-01) and Foo.2020-01-01.mp4 will be recognized as the same episode despite the fact that a special released on 2020-01-01 also exists.

Unfortunately, I don't see the handling of specials in Kodi to be changed any time soon. This functionality is deeply rooted in the Kodi core code that is written in C++ and it requires a person familiar with all the internal mechanics.

I don't know, maybe specials should get their own episode numbers in seasons outside regular episode numbers. E.g. episode 0 if a special was released before or on the same day as the first episode of a specific season, or last episode + 1 if a special was released after the last episode of the season.

@romanvm wrote:
The main problem with using date pattern for specials is that a special may be released on the same day as a regular episode. In that case a pattern Foo.YYYY-MM-DD.ext will be matched to a regular episode, meaning you may end up with duplicate items in you medialibrary. That is Foo.s01e01.mp4 (released on 2020-01-01) and Foo.2020-01-01.mp4 will be recognized as the same episode despite the fact that a special released on 2020-01-01 also exists.

Unfortunately, I don't see the handling of specials in Kodi to be changed any time soon. This functionality is deeply rooted in the Kodi core code that is written in C++ and it requires a person familiar with all the internal mechanics.

I don't know, maybe specials should get their own episode numbers in seasons outside regular episode numbers. E.g. episode 0 if a special was released before or on the same day as the first episode of a specific season, or last episode + 1 if a special was released after the last episode of the season.

I don't know anything about the internals of Kodi either, but it won't be as easy as adding another regex then ("S\d+E\d+.Special") like augur suggested?

We can theoretically of course give specials an episode number as well, but the problem would be that they'd be completely arbitrary. Particularly when there are both significant and insignificant specials in a season, our numbering would likely not match up with anyone else's.

I can think of another workaround, not particularly pretty, but it could be the best option available until Kodi can be improved: label a special "S00E20200101". The TVmaze Kodi scraper would see the "S00" part and recognize that it's a special, then see "E20201225" and use that to look up the episode (with type of "significant_special" or "insignificant_special") by the date of 2020-12-25 in the TVmaze API.

The problem, as I see it, has 2 sides:

1. How to name specials files. The workaround with a date-based name works as long as you don't have a regular episode with the same date. A better alternative would be to use episode titles for names, but unfortunately Kodi does not support title-based names for episode files. I'm a Python dev but I know a bit of C++ and currently I'm looking into this issue and have made some progress.

2. How to present specials. Historically, TheTVDB used special zero-season numeration for specials and Kodi follows this convention when presenting episodes. But TVmaze uses its own convention that probably makes more sense, that is, season numbers without episode numbers. Technically, Kodi does not require episode numbers and such items by default will be presented at the beginning of an episode list for a specific season. But you can use date-based sorting for episodes to place specials at their correct place. If it's OK, then I'll remove this zero-season number workaround for specials in the next version of the TVmaze scraper, so that specials will be presented in their respective seasons instead of a dedicated specials listing.

@romanvm wrote:
1. How to name specials files. The workaround with a date-based name works as long as you don't have a regular episode with the same date. A better alternative would be to use episode titles for names, but unfortunately Kodi does not support title-based names for episode files. I'm a Python dev but I know a bit of C++ and currently I'm looking into this issue and have made some progress.

Not sure that I follow, what do you mean with "use episode titles for names"? Do you mean to match (specials) episodes based on the episode title, instead of on number/date? I suppose that should work pretty well too, if that's easier than implementing parsing "2010-01-01.Special".

2. How to present specials. Historically, TheTVDB used special zero-season numeration for specials and Kodi follows this convention when presenting episodes. But TVmaze uses its own convention that probably makes more sense, that is, season numbers without episode numbers. Technically, Kodi does not require episode numbers and such items by default will be presented at the beginning of an episode list for a specific season. But you can use date-based sorting for episodes to place specials at their correct place. If it's OK, then I'll remove this zero-season number workaround for specials in the next version of the TVmaze scraper, so that specials will be presented in their respective seasons instead of a dedicated specials listing.

If you want to align with the decision we made on the website (which people seem to be very happy with), you could place significant specials within the actual season (sorted by date), but leave insignificant specials in a separate "season 0".