Release - Kodi Selective Cleaner addon support

Release Kodi Selective Cleaner addon support - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Add-on Support (https://forum.kodi.tv/forumdisplay.php?fid=27)
+---- Forum: Program Add-ons (https://forum.kodi.tv/forumdisplay.php?fid=151)
+---- Thread: Release Kodi Selective Cleaner addon support (/showthread.php?tid=369984)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

RE: Kodi Selective Cleaner addon support - HeresJohnny - 2023-03-29

(2023-03-29, 19:58)jbinkley60 Wrote: Overall though I am still a bit at a loss on what the musicbrainz info is really used for inside Kodi. I do see some value in the musicbrainz id being used to differentiate two albums or artists with the same name for album or artwork alignment but beyond that it is less clear..

I think the vision was to be able to display lots of info for artists, albums and songs while already browsing within Kodi but also via the OCD when playing a song. The latter is completely skin dependant. The default Estuary skin is no good example for that but presumably there are skins out there which do a much better job. I sometimes use the Estuary Mod v2 Nexus skin but haven't tried out that side of Kodi enough to know if there are really music centric skins which could visualize so far unused imported tags. That, however, should be a separate topic in the music section maybe, IMHO.

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-29

(2023-03-29, 21:16)HeresJohnny Wrote: I think the vision was to be able to display lots of info for artists, albums and songs while already browsing within Kodi but also via the OCD when playing a song. The latter is completely skin dependant. The default Estuary skin is no good example for that but presumably there are skins out there which do a much better job. I sometimes use the Estuary Mod v2 Nexus skin but haven't tried out that side of Kodi enough to know if there are really music centric skins which could visualize so far unused imported tags. That, however, should be a separate topic in the music section maybe, IMHO.

Thanks. I suspected musicbrainz is being used by addons more than by Kodi itself, given the current music database schema. I am actually going to reach out to the Mezzmo developers to see if they can pull the musicbrainz tags out of music files for Kodi uses such as you describe. Mezzmo scrapes information and artwork differently so no direct internal musicbrainz use either.. Worst case scenario I could write something to pull the musicbrainz info and insert it, similar to the Mezzmo Artwork Checker I wrote which allows regular scraping of new artwork without touching any of the underlying normal database metadata. Yep, we can move this part of the discussion to the music section.

Thanks,

Jeff

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-29

(2023-03-29, 21:07)HeresJohnny Wrote: On the performance side, the add-on is pretty fast now. After correcting all mismatches through rescraping the analyzer cleaned out hundreds of season mismatches by itself which took around 3-minutes on my oldish "server". However, one should avoid clicking in Kodi too much as the interface appears to freeze in between.

Just so I understand, it took the Kodi Selective Cleaner 3 minutes to clean a table ? If so, I will look to add a progress indicator in an upcoming release.

Thanks,

Jeff

RE: Kodi Selective Cleaner addon support - Pr.Sinister - 2023-03-30

(2023-03-29, 18:42)jbinkley60 Wrote: Out of curiosity how fast or how long did it take for the cleaning on those tables ? I would expect subsequent cleaning will not take nearly as long since it won't be building up over a long period of time.

Thanks,

Jeff

So my only suggestion would be to show the totals when just analyzing. I didn't know the amount of records to clean until AFTER the cleaning.

As for a feature once all bugs are ironed out would be to trigger an automatic cleaning (with optional backup and selection of what tables to clean) after a library update event.

For MySQL, you could create a backup of the database to a Self-Contained .sql file. If the addon is installed on the same server as the MySQL DB, you could use the built in msqldump.

When doing it through the MySQL Workbench GUI, you can see the command:

I was pleasantly surprised by the cleaning speed. I would say around 2 minutes tops for each of those 3 tables... I could be wrong but i remember saying wow that was fast so it's awesome.

Before cleaning the database with your addon, a Kodi Clean Function would take around 10 minutes. Now, it takes just under 4 minutes!

RE: Kodi Selective Cleaner addon support - Nanomani - 2023-03-30

Tested with Estuary and minimal custom addons, Kodi 20.1 windows x64 with mysql (MariaDB 10.x) on NAS Synology.
From Kodi side all works well.

But KSC addon can't connect to MYSQL.
KSC addon : v0.0.9j

"advancedsettings.xml" :
https://paste.kodi.tv/ofowacotay

Partial debug log related to KSC addon :
https://paste.kodi.tv/gawetucigi.kodi

Full debug log :
https://paste.kodi.tv/letopigava.kodi

The only information that may explain this behaviour is the mysql port that is not the default 3306 but 3307 in my config.
Thanks for any answers Smile

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-30

(2023-03-30, 01:09)Pr.Sinister Wrote: So my only suggestion would be to show the totals when just analyzing. I didn't know the amount of records to clean until AFTER the cleaning.

How about something like this ? It shows the # of records to be cleaned and the # which have data integrity errors when you run the Analyze Table function.

Thanks,

Jeff

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-30

(2023-03-30, 09:52)Nanomani Wrote: The only information that may explain this behaviour is the mysql port that is not the default 3306 but 3307 in my config.
Thanks for any answers

Thanks for reporting this and for any of you who are Simpsons fans this was my Homer Simpson moment for the day <Dooohhhh>. I totally missed passing the MySQL port information to the MySQL connector. I have uploaded version 0.0.9k which should fix this. As well I have optimized a couple of the queries to go a little faster (actor_link and file table analyzer queries) and I've added the suggestion for the counts when you run the Analyze Table function.

Please let me know if this fixes the issue you are seeing.

Thanks,

Jeff

RE: Kodi Selective Cleaner addon support - Nanomani - 2023-03-30

(2023-03-30, 12:04)jbinkley60 Wrote: I totally missed passing the MySQL port information to the MySQL connector.
I have uploaded version 0.0.9k which should fix this.
Please let me know if this fixes the issue you are seeing.

Thanks,

Jeff

Thanks for the quick fix.
I have 2 new message now :

xml:
KS Cleaner MySQL server database not found: myvideos121

xml:
KS Cleaner MySQL server database not found: mymusic82

Partial debug log :
https://paste.kodi.tv/voqoriwewi.kodi

I don't know if base name is case sensitive ?
On my side the base created by Kodi are named like this :

MyVideos121
MyMusic82

EDIT : Yes it is Case Sensitive

If you change this lines all works fine :

https://github.com/jbinkley60/Kodi-Selective-Cleaner/blob/ecb350f57726551d9b4b95d9e9d1b29bd008b158/resources/lib/common.py#L130
by

xml:
dbname = 'MyMusic' + dbver

and
https://github.com/jbinkley60/Kodi-Selective-Cleaner/blob/ecb350f57726551d9b4b95d9e9d1b29bd008b158/resources/lib/common.py#L103
by

xml:
dbname = 'MyVideos' + dbver

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-30

(2023-03-30, 12:33)Nanomani Wrote:
(2023-03-30, 12:04)jbinkley60 Wrote: I totally missed passing the MySQL port information to the MySQL connector.
I have uploaded version 0.0.9k which should fix this.
Please let me know if this fixes the issue you are seeing.

Thanks,

Jeff

Thanks for the quick fix.
I have 2 new message now :
xml:
KS Cleaner MySQL server database not found: myvideos121
xml:
KS Cleaner MySQL server database not found: mymusic82
Partial debug log :
https://paste.kodi.tv/voqoriwewi.kodi

I don't know if base name is case sensitive ?
On my side the base created by Kodi are named like this :

MyVideos121
MyMusic82

EDIT : Yes it is Case Sensitive

If you change this lines all works fine :

https://github.com/jbinkley60/Kodi-Selective-Cleaner/blob/ecb350f57726551d9b4b95d9e9d1b29bd008b158/resources/lib/common.py#L130
by
xml:
dbname = 'MyMusic' + dbver
and
https://github.com/jbinkley60/Kodi-Selective-Cleaner/blob/ecb350f57726551d9b4b95d9e9d1b29bd008b158/resources/lib/common.py#L103
by
xml:
dbname = 'MyVideos' + dbver

Ironically in my MySQL setup the databases are named myvideos121 and mymusic82 . Another way to fix this, without editing the KS Cleaner code is to add the database names to your advancedsettings.xml file. Somebody warned be before that it might be better to allow folks to hard code some of these settings in the addon settings vs. scraping them from the advancedsettings.xml file. Now I am seeing why.

I'll go with the consensus here on whether folks want me to change the default hard coding to MyVideos or myvideos etc. I am not a MySQL user so I'll go with what folks want. I could add a setting for case sensitive or not but that would only be for standard database names. That might be the compromise or I could add two settings in the addon to allow you to override the hard coding but not require an advancedsettings.xml database name. . Open to suggestions.

Thanks,

Jeff

RE: Kodi Selective Cleaner addon support - Nanomani - 2023-03-30

(2023-03-30, 13:14)jbinkley60 Wrote: Another way to fix this, without editing the KS Cleaner code is to add the database names to your advancedsettings.xml file.

I should have thought of that Smile

(2023-03-30, 13:14)jbinkley60 Wrote: Somebody warned be before that it might be better to allow folks to hard code some of these settings in the addon settings vs. scraping them from the advancedsettings.xml file. Now I am seeing why.

I agrea with that.
IMHO Default values that can be modified from the addon settings seems a good idea

(2023-03-30, 13:14)jbinkley60 Wrote: I could add two settings in the addon to allow you to override the hard coding but not require an advancedsettings.xml database name.

I vote for this.

Anyway thanks for your very promising and usefull addon Smile

RE: Kodi Selective Cleaner addon support - Pr.Sinister - 2023-03-30

Found a bug when cleaning the files table.

It seems when cleaning the file table, it removes the watched states associated with the files, even though they are still named the same and in the same directory.

I have a ton of files that do not fall under movies or tv shows and are just sitting on shares.

I go to the videos section in files mode and watch them from there. Kodi saves the resume position and the watched status in the files tables and cleaning it seems to remove those.

It has erased watched statuses that date back to at least 2010.

It's easily reproduceable as all you have to do is play a file that is not in your library and you will see it has been watched if your skin has the indicators enabled and if you return to that file after cleaning the files table, you will see watched indicator is gone.

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-30

(2023-03-30, 14:08)Pr.Sinister Wrote: Found a bug when cleaning the files table.

It seems when cleaning the file table, it removes the watched states associated with the files, even though they are still named the same and in the same directory.

I have a ton of files that do not fall under movies or tv shows and are just sitting on shares.

I go to the videos section in files mode and watch them from there. Kodi saves the resume position and the watched status in the files tables and cleaning it seems to remove those.

It has erased watched statuses that date back to at least 2010.

It's easily reproduceable as all you have to do is play a file that is not in your library and you will see it has been watched if your skin has the indicators enabled and if you return to that file after cleaning the files table, you will see watched indicator is gone.

Good catch. You are correct that right now the cleaner compares the files table to movies, episodes and musicvideos looking for a match. We have a couple of options:

Switch to only comparing if there is a path table entry that aligns to the idPath value in the files table. This would eliminate the issue but could leave orphaned file table entries for movies, musicvideos and music.
Add checking the path table to the current query but agree to a path table value to be the key.

For the second option, for the files you have which aren't aligned to the current query (i.e. not a movie, episode or music video) do the corresponding path table fields for strScraper, strContent and idParentPath have a value in them ? Those seem to be the most likely fields to match on for excluding cleaning.

Thanks,

Jeff

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-30

(2023-03-30, 13:45)Nanomani Wrote: I vote for this.

Anyway thanks for your very promising and usefull addon

Quick question, what platform are you running MySQL on, Linux ? I've coded the feature to allow an override of the defaults in the addon settings but when testing under with a Kodi client on Windows and MySQL on Windows the database names aren't case sensitive.

Jeff

RE: Kodi Selective Cleaner addon support - Nanomani - 2023-03-30

(2023-03-30, 17:02)jbinkley60 Wrote: Quick question, what platform are you running MySQL on, Linux ?

Jeff

Yes I running MySQL on Linux.

EDIT : On NAS Synology (MariaDB 10.3.x)

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-30

(2023-03-30, 17:12)Nanomani Wrote:
(2023-03-30, 17:02)jbinkley60 Wrote: Quick question, what platform are you running MySQL on, Linux ?

Jeff

Yes I running MySQL on Linux.

EDIT : On NAS Synology (MariaDB 10.3.x)

Thanks. I think we've stumbled upon the difference in case sensitivity. Anyway, I have the settings options coded and working. I'll post a new version (and update the Wiki page) either later today or tomorrow. I am waiting on some input on the file table analyzer.

Jeff