+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: General Support (https://forum.kodi.tv/forumdisplay.php?fid=111)
+---- Forum: OS independent / Other (https://forum.kodi.tv/forumdisplay.php?fid=228)
+---- Thread: Anyone come up with a solution for documentaries yet? Documentary scraper/DB? (/showthread.php?tid=333307)
Pages:
1
2
Anyone come up with a solution for documentaries yet? Documentary scraper/DB? - foetus66 - 2018-07-01
Every couple years or so, I get online and find all the recent posts on various forums for Plex and Kodi, from people who want their documentaries in one place, and I always think SURELY someone has come up with something by now. It's been the better part of a decade now though, so I thought I'd just post my own thread for once instead of sifting through the carnage of shattered dreams.
Those of us who like documentaries, and have a sizable collection because we are in the mood for a documentary sometimes, just want to simply browse documentaries. Whether it is a 90 minute feature or a 6 episode series could not possibly be less important -- we are not interested in a TV show or a movie at this time, we just want to watch a documentary. That is why we keep them stored separately in a location called 'Documentaries',which only contains documentaries, in feature length as well as series length. This location does not contain ANY movies or TV shows -- we have those in a different location, called 'Movies' and 'TV Shows', respectively. We don't want to browse 1-part documentaries in a different place from 3-part documentaries, and we DEFINITELY don't want to wade through Back to the Future and friggin' Seinfeld.
OK sorry if I started started seeming a little hostile there -- I just want to be really clear about what I mean right off the bat.
I recall the last time I tried to finagle something in Kodi, I thought I'd just set "This folder contains" to 'Movies', scan it, change it to 'TV shows' and then scan for new content individually on each folder that contains a series. I'm willing to do some extra maintenance on this particular media source. Alas, as soon as you change the type to 'TV shows', even without scanning or refreshing anything, all of the feature length documentaries immediately show up as unscraped.
I'd love to be able to choose from Movies, TV Shows, or Documentaries when setting up the source.. and honestly felt a little frustrated when Music Videos became an option. Which isn't to say I don't like having the option to scrape Music Videos as a media category -- I do. But I know a lot of us have been waiting a long time to see the Documentaries option. Heck, even if all it does is scrape one location using both TVDB and Movie Database, that would be something at least. Would REALLY love it if a Docu DB exists for it though.
I'd also probably be happy with some kind of add-on that makes it work, if anyone can point me in the right direction. I haven't had much luck finding anything.
RE: Anyone come up with a solution for documentaries yet? Documentary scraper/DB? - Karellen - 2018-07-01
(2018-07-01, 02:58)foetus66 Wrote: I'd love to be able to choose from Movies, TV Shows, or Documentariesor Sports, Cartoons, Kids Movies, Home Movies, Cinematic Universes, Foreign Movies, Anime... you get the idea.
We cannot cater for all of them, and Documentaries is no different to any of the others that I listed.
Sometimes, not everything comes out of the box ready to go. Think of "Some Assembly Required". So Kodi can do what you want, but you will need to set it up first.
Read here, but I am guessing over the years you have been given this... https://kodi.wiki/view/Custom_home_items
RE: Anyone come up with a solution for documentaries yet? Documentary scraper/DB? - foetus66 - 2018-07-01
Sorry, but documentaries are absolutely different from foreign movies, kids movies, animated movies. They are MOVIES. Home movies is not a distributed media and having a db scraper doesn't make sense. Cartoons are animated Movies or TV shows. I guess sports is the one example that does fit, although I don't think many people keep libraries of sports games, the demand surely wouldn't be there like it is for documentaries.
RE: Anyone come up with a solution for documentaries yet? Documentary scraper/DB? - pkscout - 2018-07-01
I can think of two possible things. One is easier but doesn't give you library integration, the other is harder but does.
Easy:
Create a video source that points to your folder of documentaries, but leave it assigned to NONE for the scraper and set it not to be scanned when doing library updates. After you create the source open the context menu when it's highlighted and select ADD TO FAVORITES. From now on when you want to watch a documentary just go to your favorites, select Documentaries (if that's what you called the source), and pick one from the list. This will just show you file names, but as long as you haven't named the documentaries using random characters, that could work.
Hard (or at least manually intensive):
Set the source as above but instead of None for scrapper choose Movies or TV Shows (sorry, you have to pick one) and set the information provider to LOCAL INFORMATION ONLY. Now for every single documentary you need to manually create an NFO file.
https://kodi.wiki/view/NFO_files
If you choose TV Shows, I'd suggest making every documentary have the same show name so that they show up in the same directory. For season you can use 0 (zero) and then assign each documentary a sequential episode number. If you choose Movies (which I think might be better), then in the nfo file make sure every documentary has the same set name (i.e. Documentaries). Then when they are added to the library all the documentaries will be in your Movies area under Documentaries (when listing all movies by title).
RE: Anyone come up with a solution for documentaries yet? Documentary scraper/DB? - Karellen - 2018-07-01
Wow, you do have some quite strong assumptions there based on nothing but, seemingly, your gut feeling. It does show a lack of knowledge of Kodi, Sources and Scrapers.
Anyway, I have given you the answer. It's up to you if you want to use it.
Good luck.
RE: Anyone come up with a solution for documentaries yet? Documentary scraper/DB? - Klojum - 2018-07-01
(2018-07-01, 02:58)foetus66 Wrote: But I know a lot of us have been waiting a long time to see the Documentaries option. Heck, even if all it does is scrape one location using both TVDB and Movie Database, that would be something at least. Would REALLY love it if a Docu DB exists for it though.If there are so many of you by now, what is the problem with all you starting the DocuDB.com? Apparently you all know everything there is to know about documentaries, so why are you the ones that keep waiting for others to do the work? The IMDB people, the TMDB people and the TVDB people made that initial step. Someone had to be the first man on the moon.
(2018-07-01, 04:52)foetus66 Wrote: Sorry, but documentaries are absolutely different from foreign movies, kids movies, animated movies.No, they're not. They are all the same actually. All have moving pictures in them. Just moving pictures of different things though.
RE: Anyone come up with a solution for documentaries yet? Documentary scraper/DB? - jjd-uk - 2018-07-01
(2018-07-01, 02:58)foetus66 Wrote: I recall the last time I tried to finagle something in Kodi, I thought I'd just set "This folder contains" to 'Movies', scan it, change it to 'TV shows' and then scan for new content individually on each folder that contains a series.
Can I ask why you tried this? are all documentaries not available via the TVDB? if so have you tried adding them? or is because feature length documentaries that didn’t premiere on tv aren’t allowed on there (unless something has changed)?
RE: Anyone come up with a solution for documentaries yet? Documentary scraper/DB? - foetus66 - 2018-07-01
(2018-07-01, 03:11)Karellen Wrote: Read here, but I am guessing over the years you have been given this... https://kodi.wiki/view/Custom_home_items
Thank you for the effort, I do appreciate it. I read through this (have actually been to this page a number of times before) and I may very well be missing something but I am not totally seeing how this will help.
(2018-07-01, 05:14)pkscout Wrote: Create a video source that points to your folder of documentaries, but leave it assigned to NONE for the scraper and set it not to be scanned when doing library updates. This will just show you file names, but as long as you haven't named the documentaries using random characters, that could work.
Yes, this is more-or-less what I have now, and it is what I keep coming back to when my attempts to improve it fail. My whole mission here is to be able to have all of the titles/art/descriptions populated, if not automatically, then with a reasonable amount of massaging. Everything in the folder is successfully scraped and displays nicely, but not all at the same time. It depends on which media type I have chosen, but (for example) as soon as I change it to, say, The Movie Database, the single-installment documentaries now appear while all of the data for multi-part documentaries (that was scraped from TVDB) goes away and you just see the folder name.
(2018-07-01, 05:14)pkscout Wrote: Set the source as above but instead of None for scrapper choose Movies or TV Shows (sorry, you have to pick one) and set the information provider to LOCAL INFORMATION ONLY. Now for every single documentary you need to manually create an NFO file.
https://kodi.wiki/view/NFO_files
Thank you, I have toyed with this with SOME success but have found the manual creation of files to be tedious. I would slog through it if only once, but to commit to the process for all future additions makes the method less of a long-term solution. I've looked for utilities that will scrape the folder and create the .NFO files based on the data that is downloaded, but I haven't had much luck getting that to work. It's been a couple years, maybe there's something better now. I also wondered today if it's possible to take the xml from a Video Library export and make it into an nfo file. If so, perhaps I could do one Export with the Documentaries source pointed to The TVDB, and one with it pointed to The Movie Datbase, so I can get all of the titles. I'll hopefully have a chance to mess with this before the weekend is over.
(2018-07-01, 06:17)Klojum Wrote: If there are so many of you by now, what is the problem with all you starting the DocuDB.com? Apparently you all know everything there is to know about documentaries, so why are you the ones that keep waiting for others to do the work?
Alright, I suppose I should have expected the "why don't you do it" response. Of course you're right.. And you're right that my know-it-all tone was not helpful. I let my frustration get the best of my me because if anything I thought I may have over-explained that I am not looking to subdivide types of movies. Then the first response was "we can't cater to everyone who wants to separate kids movies, foreign movies, animated movies." Argh. That's not at all what I'm talking about.
I only mentioned DocuDB as my ideal outcome to help explain what I'm after. Really, I just want to be able to store documentaries in the same place and scrape them regardless of whether they are one part long or a few. I am not a database developer, nor is DocuDB a dream I need to see realized. I have seen the idea discussed elsewhere though and given my support in the form of "that'd be awesome if you get it going, guys"
Perhaps there aren't as many people out there who want to do this as I think there are. I've just seen it come up enough times to know that I'm definitely not alone in thinking a documentary that has been broken up into a few installments is fundamentally different from ongoing television series.
Realistically I think a lot of people assume out of the gate that they will be able to catalog documentaries this way, but when they find out that it's going to be a hassle they just adapt. And that's fine. But 8 years ago on a hiking trip when I said "I wish there was a Shazam for plant identification" I had a feeling it would exist sooner or later. Now it does, in numerous forms, and I'm not surprised. I am surprised if nobody came up with a thing to combine documentaries in a scrapeable way in Kodi, but yes I put the same amount of resources into making it happen as I did into Shazam for plants: None.
(2018-07-01, 06:17)Klojum Wrote:(2018-07-01, 04:52)foetus66 Wrote: Sorry, but documentaries are absolutely different from foreign movies, kids movies, animated movies.No, they're not. They are all the same actually. All have moving pictures in them. Just moving pictures of different things though.
By that logic there is no difference between TV shows and Movies either.. and coincidentally, if they weren't different, I wouldn't be having this problem!
(2018-07-01, 10:46)jjd-uk Wrote:(2018-07-01, 02:58)foetus66 Wrote: I recall the last time I tried to finagle something in Kodi, I thought I'd just set "This folder contains" to 'Movies', scan it, change it to 'TV shows' and then scan for new content individually on each folder that contains a series.
Can I ask why you tried this? are all documentaries not available via the TVDB? if so have you tried adding them? or is because feature length documentaries that didn’t premiere on tv aren’t allowed on there (unless something has changed)?
Yes, exactly. I don't know about what's allowed per se. There are many documentaries that are not recognized by TVDB and many that are not recognized by the Movie Database. For the MOST part it is feature lengths showing up when I have The Movie Database, and multi-part documentaries showing up when I use TMDB. There have been a few interesting instances of overlap between the two though.
I've tried a few different movie scrapers in the hopes one will have a complete documentary series database in there, but have yet to find one that will scrape everything (or even most stuff) at once. That said, I haven't tried them all. If you happen to have an 'information provider' to suggest, Movie or TV, please let me know.
RE: Anyone come up with a solution for documentaries yet? Documentary scraper/DB? - jjd-uk - 2018-07-01
If you need to use a combo of movie and tv scrapers then it’s simply not possible to have a single menu entry for all Documentaries, the best you can do is have separate menu entries for feature length documentaries and series documentaries which could be facilitated by having
Documentaries (top level folder with no content set)
—— Feature Length (subfolder with content set to movie)
—— Series (subfolder with content set to tv)
Then create 2 menu entries filtering on path.
RE: Anyone come up with a solution for documentaries yet? Documentary scraper/DB? - Karellen - 2018-07-01
(2018-07-01, 11:48)foetus66 Wrote: I also wondered today if it's possible to take the xml from a Video Library export and make it into an nfo file.No need to. Just Export to Separate Files and this will create NFO Files for each video in the library... https://kodi.wiki/view/Import-export_library
(2018-07-01, 11:48)foetus66 Wrote: I only mentioned DocuDB as my ideal outcome to help explain what I'm afterLets assume that DocuDB won't be a reality in the near future.
Another option is to edit the nfo files to convert them from one type to another. There's only about 10 tags different between movie and episode nfo's. and only about 4 are critical. The rest will be ignored if they are wrong.
https://kodi.wiki/view/NFO_files/Movies#Movie_nfo_Tags
https://kodi.wiki/view/NFO_files/TV_shows#TV_Episode_Tags
RE: Anyone come up with a solution for documentaries yet? Documentary scraper/DB? - annomatik - 2018-07-07
So, well, I'd like to write a little bit about Kodi concepts, which I learned the past couple of years. Maybe that helps, maybe not, I'm not sure.
First of all "only documentaries, nothing else, seperate from anything else". Well... For me, this calls for a so called profile.
A profile is pretty much a dedicated Kodi instance. A profile can use the same sources like other Kodi profiles and e.g. just have different watched-states. You can also import completely different sources. A common example is separating Adult stuff from everything else, myself, I e.g. also have a Kids and a Fitness profile. This might already be the solution you are looking for.
Then there's the genre field, available for movies and tv shows. The distinction between movies and tv shows is a core component of Kodi. Maybe there is some sort of Addon available, which merges them, but I don't know of that and I also got used to that distinction and like it.
For Kodi, the content of the genre field does not matter. You can add as many genres to a movie or a tv show as you like. They can be free text, you can e.g. fill them via meta-data scrappers like thetvdb, moviedb or imdb. So for Kodi, "genre=documentary" is just a field, nothing more, nothing less.
Another field I like using is the "tag" field. Pretty much the same like a genre, but I usually let the scrappers fill up the genre fields and fill up the tag fields myself. I for example use them for topics. You could add a tag "Nature" for all the nature related documentaries, a tag "Consumerism" for everything related to buying stuff, and so on.
Then, there are scrappers. Like everything in Kodi, there is a separation between Movie scrappers and TV Show scrappers.
For example, one-off documentaries like "More than Honey" can be found via a movie scrapper, on IMDB: https://www.imdb.com/title/tt2263058/
On the other hand, multi-part documentaries, like "Planet Earth II" ( https://www.thetvdb.com/series/planet-earth-ii ) or multi-year documentaries like "Horizon" from BBC which has been going on since 1964 (!) ( https://www.thetvdb.com/series/horizon/ ) can be found via tv show scrappers, most prominently on thetvdb.com.
If you still insist on merging all documentaries, regardless if they are tv show and movies, you can do that, but the only way I know how is a lot of work. You can create a new profile, just for documentaries. You can create a new big folder, which contains *all* of your documentaries. And you can add a (movie) nfo file to each and every file. Then, Kodi can index all of these nfos and that's it.
Personally, in my experience ( I currently have 177 tv shows with genre=documentary and 230 movies with genre=documentary in my master profile), you can find most documentation tv shows in thetvdb.com. But not all of the "documentation movies" in e.g. imdb, as they often are very small indie productions. You can of course also add everything you like yourself to imdb.com and thetvdb.com, these platforms are open to the public. There are scrappers dedicated to anime, scrappers dedicated to porn, but I don't know about scrappers dedicated to documentaries. You can of course write your own, if you know how to write a webservice / website.
You could of course also use Kodi just as a video player. New profile, one source pointing to the directory with all your documentaries in it, that's it. But this would mean not being able to use all the "fun stuff" in Kodi.
I hope that helped a little bit.
RE: Anyone come up with a solution for documentaries yet? Documentary scraper/DB? - foetus66 - 2018-07-09
I don't really want to have NOTHING but documentaries in Kodi, I just want them to be in their own place (as I currently have them) and able to display media information. I put my three sources into "Favorites" (per suggestion from pkscout above) which tidied up my main menu to the basic three categories: Movies, TV, and Documentaries. I like this.
I then trudged through the process of creating .nfo files for EVERYTHING in the Documentaries folder, which I did by spinning up a whole new instance of Kodi on a computer I don't use for media stuff, just to be able to scrape and then export documentaries only without the rest of my library interfering (or being interfered with). I scanned with TMDB, then exported, copied all of the .nfo files that were created to a new folder, then scanned with TVDB, and exported again. Then I went through and manually checked every one, deleting the ones that didn't work (ie Planet Earth when scanned as Movies comes up as a movie from 1974, but does scrape accurately from TVDB) and made sure they are all named correctly and put in the right place.
I now have accurate .nfo files for both single-part and multiple-part documentaries -- the multi-part documentaries include a tvshow.nfo for the series, and individual .nfo files for each installment description. The single-part documentaries have a nfo that matches the filename and contain all of the information for the documentary.
Lo and behold, I have the same problem. I still have to choose "Movies" or "TV Shows" under "This directory contains" and even though I select "Local Information Only" for my information provider, it will NOT show the data from the local NFOs of multi-part documentaries when I have "Movies" selected. Likewise, it will still not show the single-part documentaries when I have TV Shows selected. If I chose "None" it just doesn't show any info at all. Again, all of the .nfo files are physically there, and accurate. I even tried selecting Music Videos and that obviously doesn't work either -- it does not show any info from the .nfo files even though I have "Local Information Only" selected.
RE: Anyone come up with a solution for documentaries yet? Documentary scraper/DB? - foetus66 - 2018-07-09
OK right after I posted, I realized the trick is to set it to TV Shows, then go into the folders for all of the single-part documentaries and simply rename the nfo that matches the video filename to tvshow.nfo instead.
I liked it better matching the filename of the video, but that is not a show stopper. Happy to have it working! Would love to see some way to export the .nfo file for just one entry (or at least for just one media source) in the future so I don't have to have a whole separate install of Kodi to create the .nfo files.
RE: Anyone come up with a solution for documentaries yet? Documentary scraper/DB? - jjd-uk - 2018-07-09
You could try experimenting with one of the Media Managers such as Ember that support Kodi compatible nfo's, might make the process simpler, see https://forum.kodi.tv/forumdisplay.php?fid=116
RE: Anyone come up with a solution for documentaries yet? Documentary scraper/DB? - wcndave - 2019-08-26
(2018-07-09, 08:45)jjd-uk Wrote: You could try experimenting with one of the Media Managers such as Ember that support Kodi compatible nfo's, might make the process simpler, see https://forum.kodi.tv/forumdisplay.php?fid=116
I think I have a lot of sympathy for the OP. This is something I'd like to do too.
The thing is that I have special interest, documentaries, music videos/documentaries, stand up comedy etc...
I don't really want to see "turning for woodworkers", and "Queen greatest hits", or "Rollings Stones Documentary", or "Wimbledon Final 2009" when I select movies or tv shows.
I do think they're different, and not the same as sub categories of films as suggested before.
On my Dune, which I want to replace, you can have layouts per directory.
From what I can see here, the options here are:
1. add them as movies and tv shows and live with it
2. turn all movies into tv shows using ember, instead of another kodi instance (however then they appear in tv shows)
3. use smart playlists, but they only support one type at a time
4. browse the "source", which doesn't show the full data, only the cover art, and doesn't work if you have multiple sources.
I can't see any option that lets you browse nicely, eg I don't mind using a folder based browsing method, and drilling down, however I'd like to see all the information there as I browse them.