Release - TMDb TV Show scraper (Python - Default Matrix Scraper)

Release TMDb TV Show scraper (Python - Default Matrix Scraper) - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Add-on Support (https://forum.kodi.tv/forumdisplay.php?fid=27)
+---- Forum: Information Providers (scrapers) (https://forum.kodi.tv/forumdisplay.php?fid=147)
+----- Forum: TV Show Scrapers (https://forum.kodi.tv/forumdisplay.php?fid=305)
+----- Thread: Release TMDb TV Show scraper (Python - Default Matrix Scraper) (/showthread.php?tid=357232)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

TMDb TV Show scraper (Python - Default Matrix Scraper) - pkscout - 2020-09-23

Please see post #699 for important information on a fix for issues where one source's settings were used for every source.

https://forum.kodi.tv/showthread.php?tid=357232&pid=3215040#pid3215040

This is the Python version of the TV show scraper for The Movie Database. There are versions available for both Kodi 18 (Leia) and Kodi 19 (Matrix), and this is the default scraper for Kodi 19. PLEASE NOTE THAT THE LAST VERSION FOR LEIA IS 1.1.34. THERE WILL BE NO FURTHER UPDATES TO THE LEIA VERSION.

For more details on using this scraper, visit the wiki page... https://kodi.wiki/view/Add-on:TMDb_TV_Shows

There are some settings if you want to adjust the behavior of the scraper.
GENERAL SETTINGS

Preferred language (default en-US)
The default language for searching and displaying results.

Preferred certification country (default en)
The content ratings to return (the scraper will return US ratings if none are found for the country you select)

Include certification prefix (default True)
Tells the scraper whether or not to include something before the rating.

Certification prefix (default Rated)
The text to prepend to the returned ratings.

Keep original title (default False)
Some TV shows have both English and original titles. Selecting this will give you the original title instead of the English one.

RATINGS SETTINGS

Default rating from (default: TMdb)
Scrape ratings for TMDb, IMDb, or Trakt and put them in the default ratings position. To scrape IMDb or Trakt ratings, the show/episode must have the IMDb ID in TMDb.

Also add IMDb ratings (default: False)
Add the IMDb ratings as well. Only enabled if IMDb is not set as the default.

Also add Trakt ratings (default: False)
Add the Trakt ratings as well. Only enabled if Trakt is not set as the default.

Also add TMDb ratings (default: True)
Add the TMDb ratings. Only enabled if TMDb is not set as the default. Note that this defaults to True because the scraper already has the TMDb ratings, so there is no "cost" to including them as secondary ratings.

FANART.TV SETTINGS
You can supplement the art provided by TMdB with art from fanart.tv by enabling the specific types you want, and you can also add your fanarttv client API key for more current results.

ADVANCED SETTINGS

Verbose debug logging (default: False)
The various APIs can return a metric crap ton of data (during testing it was not unusual to end up with a log file of over 10mb while scraping just a couple shows). Most of the time that extra info is not needed, so even with Kodi debug logging enabled the addon won't log the really long stuff. But if we need it for troubleshooting, we may ask you to enable this option. DO NOT ENABLE THIS UNLESS ASKED TO DO SO.

A couple things to note. First, the entry at themoviedb.org must have the tvdb ID to be able to get art from fanart.tv (yes, that's a lot of different site acronyms). Second, there is a limit to the amount of data that can be stored in the Kodi database regarding artwork. To ensure a show scrapes, the scraper limits the number of images you can download. Specifically, 350 fanart images and 350 of all other kind of images (including season images). My suggestion is to not enable artwork you don't need, but even if you do, the limit should still allow you access to all art except in the most extreme cases.

RE: TheMovieDB Python - TV Show scraper - pkscout - 2020-09-23

MATRIX Latest Release - March 24, 2021
1.4.2

fix for crash if Kodi calls the scraper without passing the source settings
change so that old TVDB XML scraper episodeguide URL will work to get new episodeguide from TMDb

1.4.1

fix for crash if TMDb returns no TV show results when using external ID in parsing nfo file

1.4.0

TV show trailers from YouTube now included in metadata (with option to turn it off)
option to select which YouTube addon is used to play trailers
scraper now parses IDs from TVDB and IMDB URLs for identifying show (thanks to tielis for these contributions)

1.3.14

fix for named seasons in NFO files being ignored

1.3.13

original title is now saved along with title in selected language
added option (default to true) to separate fanart from landscape art

1.3.12

writer credits now include anyone with TMDb department of Writing
changed location of cache files to special://temp/metadata.tvshows.themoviedb.org.python/cache

1.3.10

fix for issue with episode group seasons not being numbered correctly
fix for incorrect parsing of episode guide URL from XML scraper
fix for crash in certain circumstances if TMDb returns no results
fix for failed searches if show has extended characters in title
fix for changes in the data delivered by the TMDb API

1.3.2

fix for shows/episodes not loading cast correctly
fix for crash when episode has no name

1.3.1

scraper returns primary poster as the first poster from TMDb if no posters for the chosen language are available
updated debug logging

1.3.0

scraper now returns URL references for all possible art so Kodi can decide what to display based on system settings

1.2.3

fix for crash if show had no network listed on TMDb

1.2.2

change so that scraper returns English data for a show if data is missing in selected language
fixed order of certification preferences in setting

1.2.1

pull configuration for TMDb API from the API server and cache it
provide preview image for art lists (provides faster loading of small images)
delete expired cache files periodically
first version available via the Kodi repo

1.2.0

removed external module dependencies
version number change to reflect that Leia and Matrix code are no longer in sync (Leia will remain on 1.1.x, Matrix will be 1.2.x and higher)

1.1.13

added back Trakt ratings (different method to avoid crashes)
changed how images are retrieved to ensure something is returned in most cases (see post below for more details)

1.1.12

fix for crash if scraping an episode with an nfo file that doesn't have the TMDb ID in it

1.1.11

fix for source settings being ignored

1.1.10

removed Trakt support due to periodic crashes

1.1.9

added option to get ratings from Trakt and IMDb
added option to disable the text prepended to the content rating
re-arranged settings
since the code is shared between the two releases, synced version numbers

0.5.8 (Leia) and 1.1.8 (Matrix)

fix for crash if scraping a show with lots of artwork

0.5.7 (Leia) and 1.1.7 (Matrix)

added support for fanarttv client key

LEIA Latest Release - March 24, 2021
1.1.34

fix for crash if Kodi calls the scraper without passing the source settings
change so that old TVDB XML scraper episodeguide URL will work to get new episodeguide from TMDb

1.1.33

fix for crash if TMDb returns no TV show results when using external id in parsing nfo file

1.1.32

TV show trailers from YouTube now included in metadata (with option to turn it off)
option to select which YouTube addon is used to play trailers
scraper now parses IDs from TVDB and IMDB URLs for identifying shows (thanks to tielis for these contributions)

1.1.31

fix for named seasons in NFO files being ignored

1.1.30

original title is now saved along with title in selected language
added option (default to true) to separate fanart from landscape art

1.1.29

writer credits now include anyone with TMDb department of Writing
changed location of cache files to special://temp/metadata.tvshows.themoviedb.org.python/cache

1.1.26

fix for episode groupings numbering seasons incorrectly
fix for incorrect parsing of episode guide URL from XML scraper
fix for crash in certain circumstances when TMDb returns no results
fix for failed searches if show title has extended characters in it
fix for changes in data delivered by TMDb API

1.1.18

scraper returns primary poster as the first poster from TMDb if no posters for the chosen language are available
fix for shows/episodes not loading cast correctly
fix for crash when episode has no name
updated debug logging

1.1.17

fix for crash if show had no network listed on TMDb

1.1.16

change so that scraper returns English data for a show if data is missing in selected language
fixed order of certification preferences in setting
fixed crash if logging data with extended characters

1.1.15

pull configuration for TMDb API from the API server and cache it
delete expired cache files periodically
first version available via the Kodi repo

1.1.14

removed external module dependencies
version number change to reflect that Leia and Matrix code are no longer in sync (Leia will remain on 1.1.x, Matrix will be 1.2.x and higher)

1.1.13

added back Trakt ratings (different method to avoid crashes)
changed how images are retrieved to ensure something is returned in most cases (see post below for more details)

1.1.12

fix for crash if scraping an episode with an nfo file that doesn't have the TMDb ID in it

1.1.11

fix for source settings being ignored

1.1.10

removed Trakt support due to periodic crashes

1.1.9

added option to get ratings from Trakt and IMDb
added option to disable the text prepended to the content rating
re-arranged settings
since the code is shared between the two releases, synced version numbers

0.5.8

fix for crash if scraping a show with lots of artwork

0.5.7

added support for fanarttv client key

RE: TheMovieDB Python - TV Show scraper - Karellen - 2020-09-23

reserved

[split] TheMovieDB - TV Show scraper - bobandre - 2020-09-27

(2020-09-27, 10:13)Karellen Wrote:
(2020-09-27, 10:06)bobandre Wrote: Yes, MySQL
Ok, it is a know problem and it affects some other shows like Game of Thrones and some movies as well. See here... https://github.com/xbmc/xbmc/issues/15768

Your options are...
1. Make the change as described in the above link
2. Install the brand new TheMovieDB scraper created by @pkscout and use it as it overcomes the problem. It is not in the repo yet, so you need to download and install via zip... https://forum.kodi.tv/showthread.php?tid=357232
I know this scraper works as The Simpsons was the show we were using during the testing phase last week

only english lang with @pkscout scraper? i set setings to ru-RU, but import in Eng...

RE: TheMovieDB - TV Show scraper - Karellen - 2020-09-27

@bobandre

Thanks for reporting, and I can confirm the problem.

I have raised an issue report here... https://github.com/pkscout/metadata.tvshows.themoviedb.org.python/issues/15

You can monitor the progress of the fix there. Hopefully it won't be too long for the fix.

Also I will move these two posts to the correct thread as we are now discussing a different scraper to the one in this thread.

RE: TheMovieDB Python - TV Show scraper - pkscout - 2020-09-27

@bobandre sorry about that. I was unclear how settings are handled for scrapers, so I was reading them from the wrong place. I've got an update for Karellen to test, and once we've confirmed the fixes work, I'll push out an update. Thanks for reporting this.

RE: TheMovieDB Python - TV Show scraper - pkscout - 2020-09-27

Updated version available to fix the issue of the language (and other) source settings being ignored.

https://github.com/pkscout/metadata.tvshows.themoviedb.org.python/releases

RE: TheMovieDB Python - TV Show scraper - pkscout - 2020-09-30

With version 1.1.13 there is a change to how images are retrieved and the order in which they are displayed. Most people will probably never notice, but at some point someone is going to say that the order in Kodi doesn't match the order from TMDb, so I thought I'd go ahead and explain why. '-)

Here's the logic for images (no matter the type):

Get all images that are in the preferred language, have no language, or have English as the language.
If the most popular image has no language, keep it in the first position.
With the given exception in #2, return images in the following order: preferred language, no language, English.

Why? Many show images (posters and banners especially) are in English and have few if any translations. If you only do preferred language and no language, you often will get no art for a show. But if you add English then you'll get the most popular image by default (which is often an English one) even if you pick a different preferred language. The return order outlined above gives you (IMHO) the best mix. If the most popular image has no language, you get that. If not, then you get the most popular image in your preferred language (if available). Otherwise you get the most popular English image. And the list of other images prioritizes your language choice first.

RE: TheMovieDB Python - TV Show scraper - Karellen - 2020-10-14

(2020-10-06, 19:08)ramis52 Wrote: Hi,

i just switched over from thetvdb to tmdb scraper. is there any option in tmdb scraper to set a fallback language like in ttvb scraper? Maybe it is already set in the Background to en-us when there are no Information in the prefered Language available.
My question is because if i set the Language to de-de and the scraper didnt find any Translation for German, it will add the Plot in english. thats great But the Title of the Episode is Episode1 Episode2. Would be great if the Title will then Fallback to English too instead of adding it as Episode1 episode2 and so on. Dunno if its a bug or if its a fail on my side

regards
r

After quite a bit of work @pkscout has found a way to work around the problem at TheMovieDB.

The fix has been implemented in the new Python version of the scraper, not the XML version you are currently using.

Could you please download and install via zip this test build here and let us know if it is working correctly for you... https://github.com/xbmc/metadata.tvshows.themoviedb.org.python/issues/18#issuecomment-707891646

In case you don't know how to download a zip file. Step 1 is by clicking on the above link.

RE: TheMovieDB Python - TV Show scraper - ramis52 - 2020-10-14

awesome work! i can confirm that all my problematic shows now scrapes fine. If no DE Title of an episode is available it fallback to EN titles.

thanks alot for the fix Smile

RE: TheMovieDB Python - TV Show scraper - pkscout - 2020-10-14

(2020-10-14, 17:18)ramis52 Wrote: awesome work! i can confirm that all my problematic shows now scrapes fine. If no DE Title of an episode is available it fallback to EN titles.

thanks alot for the fix

Thanks for the testing and feedback. I have one more change to make before submitting this update to the official repo, so it'll probably be later this week. Because you installed this test on Matrix from a zip file, you may have to manually tell Kodi to update the addon when the update comes out (there have been some changes to how we handle addons installed from zip files with matrix Alpha2). But after that one manual update, you'll be back to getting the updates automatically after that.

RE: TheMovieDB Python - TV Show scraper - macomeau - 2020-10-14

I've been trying this out today, and have run into an issue with some shows not scraping. My system is English. All the shows that fail for me are non-English, but not all non-English shows fail. Some Japanese shows scrape, some don't. One French show fails, but another doesn't. A German and a Danish show scrape. An example log is here. The first show, Mobile Suit Gundam 0080, fails to scrape. The second show, Mobile Suit Gundam: The 08th MS Team, scrapes successfully. I tried both multiple times before turning on logging, and the fail shows always fail; the pass shows always pass.

The movie version had no failures on a multitude of English and non-English titles.

RE: TheMovieDB Python - TV Show scraper - pkscout - 2020-10-14

(2020-10-14, 21:36)macomeau Wrote: I've been trying this out today, and have run into an issue with some shows not scraping. My system is English. All the shows that fail for me are non-English, but not all non-English shows fail. Some Japanese shows scrape, some don't. One French show fails, but another doesn't. A German and a Danish show scrape. An example log is here. The first show, Mobile Suit Gundam 0080, fails to scrape. The second show, Mobile Suit Gundam: The 08th MS Team, scrapes successfully. I tried both multiple times before turning on logging, and the fail shows always fail; the pass shows always pass.

The movie version had no failures on a multitude of English and non-English titles.

I'm pretty sure I see where the problem is. It looks like it a show has no network that it often won't scrape. It's a pretty easy fix, so I'll try to get to it today and at least have something you can download manually to address this in the interim until the updated version gets pushed to the repo.

RE: TheMovieDB Python - TV Show scraper - pkscout - 2020-10-14

@macomeau I was right, it was an easy fix. This will by pushed out with the next release, but I don't yet know for sure when that will be. In the meantime, you can manually install this version for Leia (that's what your log should you were using) that should resolve your issue:

https://github.com/xbmc/metadata.tvshows.themoviedb.org.python/archive/leia.zip

If anyone is on Matrix and having problems with a show not scraping because they have no network listed, you can download the Matrix version of this patch at:

https://github.com/xbmc/metadata.tvshows.themoviedb.org.python/archive/matrix.zip

RE: TheMovieDB Python - TV Show scraper - macomeau - 2020-10-15

(2020-10-14, 23:19)pkscout Wrote: I'm pretty sure I see where the problem is. It looks like it a show has no network that it often won't scrape. It's a pretty easy fix, so I'll try to get to it today and at least have something you can download manually to address this in the interim until the updated version gets pushed to the repo.

Yeah, that got all but the French show. The directory I have it in is spelled in French with an accent, Résistance. If I switch the spelling to Resistance, it scrapes. Log file. The movie scraper doesn't have this problem, as far as I can tell, having found the movie Deux enfoirés à Saint-Tropez just fine.