Kodi Community Forum
Release Kodi Selective Cleaner addon support - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Add-on Support (https://forum.kodi.tv/forumdisplay.php?fid=27)
+---- Forum: Program Add-ons (https://forum.kodi.tv/forumdisplay.php?fid=151)
+---- Thread: Release Kodi Selective Cleaner addon support (/showthread.php?tid=369984)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-04-04

Everyone,

I am planning to move the Kodi Selective Cleaner to a 1.0 production release next week, if there are no further bug reports.  Note that the download and Wiki URLs will be changing for 2 reasons. First the MySQL test version has been on a test URL vs. the main addon URL. Second, I plan to move the GitHub naming to a standard Kodi addon naming convention plugin.program.kscleaner .  This will make it easier for folks who integrate and track third party addons which don't exist in the normal Kodi repository.   I'll provide an update here on the new URL and such next week.

Thanks for your continued support and I hope you find this addon useful.


Jeff

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-04-29

(2023-04-04, 11:07)jbinkley60 Wrote: Everyone,

I am planning to move the Kodi Selective Cleaner to a 1.0 production release next week, if there are no further bug reports.  Note that the download and Wiki URLs will be changing for 2 reasons. First the MySQL test version has been on a test URL vs. the main addon URL. Second, I plan to move the GitHub naming to a standard Kodi addon naming convention plugin.program.kscleaner .  This will make it easier for folks who integrate and track third party addons which don't exist in the normal Kodi repository.   I'll provide an update here on the new URL and such next week.

Thanks for your continued support and I hope you find this addon useful.


Jeff


Now that the Kodi forum is back up I will be making the changes mentioned above this week.  I've been a bit busy on a couple of other projects so nothing new has been going on with this addon in the past month.  I am interested in the next set of feature requests.  I have continued to look at the music database to potentially build an analyzer module.   As well as maybe something for the textures database.  Feedback is welcome.


Thanks,

Jeff

RE: Kodi Selective Cleaner addon support - Pr.Sinister - 2023-05-01

Encountered a major bug. Still trying to figure out the logic but here is what i have observed so maybe you can figure it out...

When i watch an episode of a TV Show, i have a service that checks if i have watched the previous episode or if it even exists in my database.

After using 0.0.9k MySQL with Strict enabled, i started getting warnings on multiple shows that previous episode was missing entirely from the database.

Upon investigating, they were indeed missing from the database but were present in my file storage. Furthermore, the episode count displayed by Kodi was accurate but upon entering the season, the amount of episodes listed was different.

Image

The image above shows that Season 10 has 12 Episodes, with 2 of them Unwatched.

Image

Here you can see the 2nd to last episode is episode 8

Image

And here you see Episodes 9 through 11 are missing. The total number of episodes is 9 with only 1 unwatched.

Image

The history inside Sonarr tells me that Sonarr downloaded all episodes one by one from newsgroups and on April 17th, i replaced episodes 1 through 8 with a version i manually downloaded straight from Prime Video with StreamFab.

Some time between April 20th and April 26th, i ran KSC and it hosed a shit ton of TV Shows. April 27th, new episode was added by Sonaar without issue. I can't tell you exactly how many of my 1937 TV Shows with ~90,000 episodes it has broken but i have found that every TV Show that has had a new episode added after the cleanup was affected (the ones i watch as the episodes air)

IMO, it has to do with Kodi keeping the original filename in the database even if you delete that file and replace it with a differently named one. 

I replace the "scene" episodes with the one from StreamFab because StreamFab lets me download them in H265 when most of the scene ones are in H264 or in HEVC that has an unnecessarily high bitrate. So it has to do with the file naming i am assuming because episodes 1-8 were in H264 and i do put the name of the codec in my naming template and i replaced them with H265 versions. Of course this is an assumption but that is what i have observed from all the corrupt shows. The ones that had episodes replaced are still here while the ones that only had 1 version of the file disappeared.

Updating the library or right-click a specific show and scanning for new content does nothing. The ONLY way i was able to get the missing episodes back into the library was by using the Refresh function after pressing i to bring up the TV Show information and then scanning for new content again.

Needless to say i am not going to do that for 1937 TV Shows. So i setup a portable Kodi with a named (not default) MySQL database and rescanned the entire ~90,000 episodes and ~9000 Movies. It's time consuming so i didn't want be without a database for the stuff i still watch but never replaced files. That's the reason for not refreshing the whole current library.

P.S., after backing up the database once at the beginning of this addon's 1st release, i never did again... Lesson learned: Don't be lazy  Big Grin

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-05-01

(2023-05-01, 16:50)Pr.Sinister Wrote: After using 0.0.9k MySQL with Strict enabled, i started getting warnings on multiple shows that previous episode was missing entirely from the database.

IMO, it has to do with Kodi keeping the original filename in the database even if you delete that file and replace it with a differently named one. 

You definitely want strict to be disabled, which is the default.  I mentioned it here.  When strict is enabled  it is much more aggressive about missing files.  See the note at the bottom:

Note 1 - Use files table strict mode carefully. Strict mode cleaning will remove file table entries that do not match against the movie, episode or musicvideo tables. This can be helpful in removing Internet downloaded and other temporary content but can also remove tile table entries / resume points for content added to the file table by other addons. Please review the analyze table contents carefully before cleaning in strict mode.

Here's the difference in the analysis:

Files - Compares unmatched file table entries against path
Files (strict) - Compares unmatched entries against episode, movie and musicvideo and streamdetails tables (audio and video separately)

You can also look in the logs and KS Cleaner should tell you why something is removed.  Lastly, if you use the analyze function in advance it will tell you what it would remove prior to running a clean function. 

Keeping the file name the same might be part of the problem but the KS Cleaner uses the indexes and not the names to match.  So if something exists in the file table but it doesn't have a corresponding file ID value in the matching table  criteria (depending on checking mode) it is considered orphaned.  If corresponding path table entries aren't being created, that is a bigger issue.  Running the file table analyze in normal mode would show that.

The challenge for the file table to find a middle group of removing what is orphaned but not those which are temporary, is the sources that Kodi uses.  They vary by addon and Internet source.  I can get away with this on Mezzmo because everything is sourced from Mezzmo so I can compare what is in the Mezzmo against the Kodi database.  Anything in Kodi which isn't in Mezzmo is temporary.  But for most folks, they have variable sources which is why normal and strict exist now.

I'd also suggest using version 0.0.9m .


Thanks,

Jeff

RE: Kodi Selective Cleaner addon support - Pr.Sinister - 2023-05-01

re-reading the stuff about strict, i realize i should have set it to NOT use strict. The forum was down so i could not check what you had said. And now i think it may be that it deleted episodes that were unwatched? Is that the case?

EDIT: Nevermind... you replied as i was typing this...

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-05-01

(2023-05-01, 17:27)Pr.Sinister Wrote: re-reading the stuff about strict, i realize i should have set it to NOT use strict. The forum was down so i could not check what you had said. And now i think it may be that it deleted episodes that were unwatched? Is that the case?

EDIT: Nevermind... you replied as i was typing this...

Ok.  Let me know if there is anything I can do to help.


Thanks,

Jeff

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-05-02

Version 1.0.0 of the Kodi Selective Cleaner has been released.  It is identical to version 0.0.9m other than incrementing the version number. 

If you were running the prior production 0.0.9 release of Kodi Selective Cleaner then there are 2 major enhancements with this release:
  • Added MySQL database support
  • Added strict and normal file table analysis

Note that the MySQL test version 0.0.9m has been pulled down and Kodi Selective Cleaner is now on a new GitHub URL to align to Kodi naming standards.

The Wiki page has also moved and these are the permanent homes now for these links.  The old links appear to redirect to the new links but I would suggest bookmarking the new links just in case the old ones stop redirecting after a certain period of time. 

I'll be spending some time to update all of the Wiki embedded links.  If you find anything now working, please let me know.

I'd like to thank those of you who provided feedback, did testing and worked with me to get this addon to production.  Hopefully it will provide the value you were seeking and we can add some cool new features in the future.


Thanks,

Jeff

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-05-25

Version 1.0.1 of the Kodi Selective Cleaner has been released on GitHub.  It has a minor bug fix related to MySQL named music databases causing an exception error.

version 1.0.1 download link



Note that the MySQL test version 0.0.9m has been pulled down and Kodi Selective Cleaner is now on the new GitHub URL to align to Kodi naming standards.

The Wiki page has also moved and these are the permanent homes now for these links.  The old links appear to redirect to the new links but I would suggest bookmarking the new links just in case the old ones stop redirecting after a certain period of time. 

I'll be spending some time to update all of the Wiki embedded links.  If you find anything now working, please let me know.

I'd like to thank those of you who provided feedback, did testing and worked with me to get this addon to production.  Hopefully it will provide the value you were seeking and we can add some cool new features in the future.


Thanks,

Jeff

RE: Kodi Selective Cleaner addon support - HeresJohnny - 2023-05-25

Thanks Jeff! Could you make an action on Github so the versions appear under "Releases"?

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-05-25

(2023-05-25, 22:08)HeresJohnny Wrote: Thanks Jeff! Could you make an action on Github so the versions appear under "Releases"?

I'll put it on my To-Do list.


Thanks,

Jeff

RE: Kodi Selective Cleaner addon support - RuralHunter - 2023-05-28

Code:
2023-05-28 14:51:55.232 T:3496    ERROR <general>: SMBDirectory->GetDirectory: Unable to open directory : 'smb://192.168.166.1/NAS/Video/TVShow/....'
                                                   unix_err:'2' error : 'No such file or directory'
Those files are deleted. But when I run video database cleanup, Kodi reports these errors. Can this addon fix this to remove those obsolete file records?

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-05-28

(2023-05-28, 14:44)RuralHunter Wrote:
Code:
2023-05-28 14:51:55.232 T:3496    ERROR <general>: SMBDirectory->GetDirectory: Unable to open directory : 'smb://192.168.166.1/NAS/Video/TVShow/....'
                                                   unix_err:'2' error : 'No such file or directory'
Those files are deleted. But when I run video database cleanup, Kodi reports these errors. Can this addon fix this to remove those obsolete file records?

No, the tool doesn't address these types of issues.  I believe this is a Kodi Sources issue where the source needs to be removed.  Someone more familiar with sources and Kodi SMB mapping should be able to assist. 


Thanks,

Jeff

RE: Kodi Selective Cleaner addon support - izprtxqkft - 2023-05-28

that usually happens when you havent removed a source from kodi properly

the 'proper' way would be to change a source's content to None then kodi will prompt to remove all its items from the library to which you answer in the affirmative and it will remove them from the database

then after that you can delete rename or otherwise make unavailable to kodi any sources

not doing the first step usually leaves orphans that clean library doesnt know how to handle

RE: Kodi Selective Cleaner addon support - RuralHunter - 2023-05-30

(2023-05-28, 15:24)jbinkley60 Wrote: No, the tool doesn't address these types of issues.  I believe this is a Kodi Sources issue where the source needs to be removed.  Someone more familiar with sources and Kodi SMB mapping should be able to assist. 


Thanks,

Jeff
OK. Thanks for the info.

RE: Kodi Selective Cleaner addon support - RuralHunter - 2023-05-30

(2023-05-28, 16:37)jepsizofye Wrote: that usually happens when you havent removed a source from kodi properly

the 'proper' way would be to change a source's content to None then kodi will prompt to remove all its items from the library to which you answer in the affirmative and it will remove them from the database

then after that you can delete rename or otherwise make unavailable to kodi any sources

not doing the first step usually leaves orphans that clean library doesnt know how to handle
No. I didn't remove the source, but just deleted those movie files in the source directory.