Release - Kodi Selective Cleaner addon support

Release Kodi Selective Cleaner addon support - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Add-on Support (https://forum.kodi.tv/forumdisplay.php?fid=27)
+---- Forum: Program Add-ons (https://forum.kodi.tv/forumdisplay.php?fid=151)
+---- Thread: Release Kodi Selective Cleaner addon support (/showthread.php?tid=369984)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2024-06-16

(2024-06-15, 22:33)kittmaster Wrote: 2 - Duplicate video detection >> This one has to be handled very carefully. Example Aliens (1986), Aliens (The Cocoon Cut) (1986), Aliens (Director's Cut) (1986), when parsed, it can't stop at just "Aliens" and say oh, there are 3 and nuke 2. So that is just an example.

I am not looking to code a fuzzy logic or like matching right now. It would need to be exact matches. Also, I am not thinking about automatic remediation just identification and let folks hand delete duplicates I sure hope folks don't have lots of duplicates. That would be another discussion for automatic remediation.

(2024-06-15, 22:33)kittmaster Wrote: 4 - NFO file creator - This will need to have conditional logic checking. For those that don't have NFO's, build as you/they need/see fit... otherwise, like myself, I use Media Companion that is entirely NFO driven to import all data into Kodi an don't want any modifications made to the file by any extension externally since Media Companion is the Master to control all Meta for Kodi import.

I am thinking it would be fully manual with an option to enable / disable the feature and a safety check to ask whether you want to over write an existing NFO file. I personally don't use them but realize that they are important to native Kodi database usage and scrapers.

Thanks,

Jeff

RE: Kodi Selective Cleaner addon support - kittmaster - 2024-06-16

(2024-06-16, 01:59)jbinkley60 Wrote:
(2024-06-15, 22:33)kittmaster Wrote: 2 - Duplicate video detection >> This one has to be handled very carefully. Example Aliens (1986), Aliens (The Cocoon Cut) (1986), Aliens (Director's Cut) (1986), when parsed, it can't stop at just "Aliens" and say oh, there are 3 and nuke 2. So that is just an example.

I am not looking to code a fuzzy logic or like matching right now. It would need to be exact matches. Also, I am not thinking about automatic remediation just identification and let folks hand delete duplicates I sure hope folks don't have lots of duplicates. That would be another discussion for automatic remediation.

(2024-06-15, 22:33)kittmaster Wrote: 4 - NFO file creator - This will need to have conditional logic checking. For those that don't have NFO's, build as you/they need/see fit... otherwise, like myself, I use Media Companion that is entirely NFO driven to import all data into Kodi an don't want any modifications made to the file by any extension externally since Media Companion is the Master to control all Meta for Kodi import.

I am thinking it would be fully manual with an option to enable / disable the feature and a safety check to ask whether you want to over write an existing NFO file. I personally don't use them but realize that they are important to native Kodi database usage and scrapers.

Thanks,

Jeff

If the match has to be exact, then there should be no issue, as long as partials aren't returned i.e. because it was "aliens" (ignore the rest) then it would be messy... but since you are stating it would have to be fully matching to qualify, then no issue exists with my given example and others like it.

I dare throw a monkey in the wrench for the NFO....... enable / disable / merge with existing?

I realize this is morphing into major feature creep, but if I didn't ask, I'm sure anyone with an keen eye would.

RE: Kodi Selective Cleaner addon support - HeresJohnny - 2024-06-16

An .nfo file creation feature already exists in the Medadata Editor add-on. Also, it exists in tools like Sonarr etc.

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2024-06-16

(2024-06-16, 16:52)HeresJohnny Wrote: An .nfo file creation feature already exists in the Medadata Editor add-on. Also, it exists in tools like Sonarr etc.

Yep. I think the requestor was looking for a context menu driven NFO creator. I've asked them to comment here on the needs and goal of the request.

Thanks,

Jeff

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2024-06-18

I decided to take the requests one at a time and have begun working on the Duplicate video detection feature. The initial menus are done and I am finishing up the SQL queries to identify duplicates. I should have an initial test release ready by the weekend. One thing I am noticing is that while movies will typically have somewhat unique titles, TV episodes can be the same title across TV show series. I am thinking for those to not only look for duplicate episodes by title but also within the same TV series. This would be in addition to looking for duplicate file names and avoid misidentifying duplicates across TV series..

I'd still like some feedback on the output format. To start with I plan to do a screen display. so folks can see the duplicates and information provided. Then we can discuss maybe a text file output option. Initially I am planning to provide the duplicate title or file name, TV series (for episodes) and path information. I may have to truncate the screen display to avoid wrapping or place each field on its own line with a space between duplicates. I am open to suggestions.

Thanks,

Jeff

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2024-06-19

I've posted a new version 1.0.6a test release of the Kodi Selective Cleaner to GitHub. This version has the beginnings of duplicate movie title checking. When you click on Video DB Data Analyzer you will see a new menu option called Duplicate Media Analysis. Clicking on this will provide you with 3 options: Movies, Episodes and Music Videos. Only Movies is working right now and this will perform a duplicate check by title and then provide a text output to the screen.

Overall it is quick. Testing on both MySQL and SQLite with databases of around 5k movies the results are displayed within a couple of seconds. Three pieces of data are returned for each duplicate: The Title, the Kodi Pat and the Kodi file name. Duplicates can be more than 2 of the same titles. You could have say 10 movies with the same title. I'll add movie File name duplicate checking once we agree on the output format.

Here's a current sample out.

I am also thinking of an addon setting to either send the output to the screen, to a text file or both.

As always, feedback is welcome.

Jeff

RE: Kodi Selective Cleaner addon support - Karellen - 2024-06-20

(2024-06-19, 21:38)jbinkley60 Wrote: I've posted a new version 1.0.6a test release of the Kodi Selective Cleaner to GitHub.

Thanks. I have had a play with this addon and I was able to find a few issues in my own library. Mainly unmatched ratings which were easily fixed.
The Seasons report is a bit of a red herring. Because the scrapers download the full seasons list and add it to the Seasons table, the report shows a lot of errors if there are no episodes in the seasons. It might confuse some users. It had me confused for a few minutes until I realised these were shows where I did not have all the episodes.

Tested the duplicates report which offers up a list of similarly named titles and found no problem.
Some might say you should filter out differences, like different (year) or ignore versions, but I think that might be too difficult?
Offer up the complete report as is then let the user decide if they need to fix anything.

Thanks for the great work Smile

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2024-06-20

(2024-06-20, 01:10)Karellen Wrote:
(2024-06-19, 21:38)jbinkley60 Wrote: I've posted a new version 1.0.6a test release of the Kodi Selective Cleaner to GitHub.
Thanks. I have had a play with this addon and I was able to find a few issues in my own library. Mainly unmatched ratings which were easily fixed.
The Seasons report is a bit of a red herring. Because the scrapers download the full seasons list and add it to the Seasons table, the report shows a lot of errors if there are no episodes in the seasons. It might confuse some users. It had me confused for a few minutes until I realised these were shows where I did not have all the episodes.

Tested the duplicates report which offers up a list of similarly named titles and found no problem.
Some might say you should filter out differences, like different (year) or ignore versions, but I think that might be too difficult?
Offer up the complete report as is then let the user decide if they need to fix anything.

Thanks for the great work

Thanks for the feedback and testing. I am open to suggestions for the seasons table. I've never had the issue in my setup because the Mezzmo Kodi addon only inserts a season when there is an episode. I didn't realize Kodi would insert a season without episodes. That's good to know.

With regards to the duplicate checking my plan is just as you say, present all the information and let the user decide. The problem with dupes is how do you know which one is correct and which one should be deleted ? I leave that to the user. I need to add some error handling to account for situations where a library item might be missing a field in the database for some reason and the addon would try to print it but it doesn't exist. If the user runs the database analyzer and cleans it ahead of dupe checking then that can't occur. But I can't rely on the order of operation by the user. That's some minor tweaking to address.

Jeff

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2024-06-20

I've posted a new version 1.0.6b test release of the Kodi Selective Cleaner to GitHub. This version has the duplicate media checking now working for all Kodi video media categories: movies, episodes and music videos. There is a new addon setting to specify the output when doing duplicate media analysis. The options are off, GUI, file and both. Off removes the option from the menus so duplicate checking is unavailable. GUI is the default. File writes the output to a text file.

Here's a complete set of analysis available in this release:

Duplicate movies by title
Duplicate movies by filename
Duplicate episodes by filename
Duplicate music videos by title
Duplicate music videos by filename

I've held off on duplicate episodes by title since I believe folks would also want that to compare against the TV Show name, which is likely going to require multiple SQL queries and such. I'd like feedback on whether episodes by filename is enough. Hopefully you find this new functionality useful and enlightening into what is actually in your video libraries.

I still have a bit of cleanup to do on the code, some improved logging and such but things should be fully functional and you shouldn't get errors.

As always, feedback is welcome.

Jeff

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2024-06-22

I've posted a new version 1.0.6c test release of the Kodi Selective Cleaner to GitHub. This version adds checking for duplicate episodes by a combination of the TV Show name, Season and Episode, so truly a duplicate episode. He's an example screenshot of the information provided to help you determine which of the duplicates you want to delete..

This is the last test version of the duplicate media analyzer feature unless bugs are found or major enhancements requested. I've added error checking and proper normal / debug logging. Note that the category selection type dialog box is multi-select so you can select Movies, Episodes and Music Videos individually or in any combination you want.

I appreciate testing and feedback.

Thanks,

Jeff

RE: Kodi Selective Cleaner addon support - HeresJohnny - 2024-06-22

Good job! I could have sworn I had no duplicates in my collection but your add-on found quite a few tv episodes that got duplicated through automatic renaming.

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2024-06-22

(2024-06-22, 19:49)HeresJohnny Wrote: Good job! I could have sworn I had no duplicates in my collection but your add-on found quite a few tv episodes that got duplicated through automatic renaming.

I am still using your old MySQL export to test with. I thought you might find some duplicates. Smile

I have to manually simulate SQLite duplicates because the Mezzmo Kodi addon will never enter a dupe into the Kodi database. Your export is very handy for testing MySQL.

Out of curiosity which output format do you prefer, GUI or text file ? If you select all 3 categories how fast does it run ? I was thinking of adding a log message which records the analysis run time, just in case folks have slow systems / databases. I am probably over thinking it.

Thanks for testing and the feedback,

Jeff

RE: Kodi Selective Cleaner addon support - ManilaByNight - 2024-06-23

Good job! It appears to have caught all the duplicates in my Movies. I never thought I would have duplicates but after running your add-on, I was surprised to find some episodes that did not match any dupes. I wonder why?

Kudos to you for this add-on. It is a BIG help indeed.
Good Luck!

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2024-06-23

(2024-06-23, 11:20)ManilaByNight Wrote: Good job! It appears to have caught all the duplicates in my Movies. I never thought I would have duplicates but after running your add-on, I was surprised to find some.

Good Luck!

I am glad it is working for you. Since this was your feature request, are you satisfied with the outcome or have any other enhancement requests ? I am pretty pleased with it so far. In the next release I have a couple of minor tweaks on the episode by file duplicate checking. The next release will provide a little more information with regards to the season, episode and TV show name.

I am not surprised that you found duplicates. I don't believe any of the Kodi scrapers check for this condition, which would be pretty normal for a scraper.

Thanks,

Jeff

RE: Kodi Selective Cleaner addon support - ManilaByNight - 2024-06-23

(2024-06-23, 13:16)jbinkley60 Wrote:
(2024-06-23, 11:20)ManilaByNight Wrote: Good job! It appears to have caught all the duplicates in my Movies. I never thought I would have duplicates but after running your add-on, I was surprised to find some.

Good Luck!

I am glad it is working for you. Since this was your feature request, are you satisfied with the outcome or have any other enhancement requests ? I am pretty pleased with it so far. In the next release I have a couple of minor tweaks on the episode by file duplicate checking. The next release will provide a little more information with regards to the season, episode and TV show name.

I am not surprised that you found duplicates. I don't believe any of the Kodi scrapers check for this condition, which would be pretty normal for a scraper.

Thanks,

Jeff

Well, I'm pretty happy with it so far fr the movies but if you have any other minor tweaks to improve it then I am all for it.

BTW, I tried it also with my TV episodes and it found some dupes which I was never aware of.

However, I noticed that it listed some episodes that did not match any dupes. I wonder why?

Kudos to you for this add-on. It is a BIG help indeed.