Release - Kodi Selective Cleaner addon support

Release Kodi Selective Cleaner addon support - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Add-on Support (https://forum.kodi.tv/forumdisplay.php?fid=27)
+---- Forum: Program Add-ons (https://forum.kodi.tv/forumdisplay.php?fid=151)
+---- Thread: Release Kodi Selective Cleaner addon support (/showthread.php?tid=369984)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

RE: Kodi Selective Cleaner addon support - gibxxi - 2023-03-25

Forgive me if I'm being stupid here Jeff, but I don't see the difference between the two ampersands or the options as you've posted them? The whitespaces added either side were added as a contingency as the artists this affects, do have two clear whitespaces either side of the ampersand, but I also know it's a special character used in XML, so in code needs to be 'escaped' for some XML (probably all - to be standards-compliant) parsers.

It's never caused an issue with Kodi core, so I went with it, but I always knew at some point it would break something, and now it's finally happened, lol.

Thanks for verifying the root cause in any case. I could of commented that section out and tried to run without the 'Separators' section, but to do that would of required me to also temporarily disable library scanning to prevent the affected artists being re-scraped again as a single artist, so I played it safe.

Many thanks, I'll download the updated 0.0.9i in a bit and try it out and edit this post if it proves to be working. My setup is nothing, if not quirky.

Wink

Dan / Gib.

EDIT: Unfortunately Jeff, It's still not working. Now I get this error:

Code:
2023-03-25 21:52:15.362 T:15768   error <general>: EXCEPTION IN (D:\Kodi Media Center\portable_data\addons\plugin.program.kscleaner\resources\lib\common.py, LINE 99 "dbname = dbname.txt + dbver"): 'xml.etree.ElementTree.Element' object has no attribute 'txt'

2023-03-25 21:52:15.362 T:15768    info <general>: There is a problem with your MySQL configuration

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-26

(2023-03-25, 23:30)gibxxi Wrote: Forgive me if I'm being stupid here Jeff, but I don't see the difference between the two ampersands or the options as you've posted them? The whitespaces added either side were added as a contingency as the artists this affects, do have two clear whitespaces either side of the ampersand, but I also know it's a special character used in XML, so in code needs to be 'escaped' for some XML (probably all - to be standards-compliant) parsers.

No stupidity here at all. I missed that in the second example the forum website I converted it to an &. I hope my update to the post makes sense. Basically instead of an & to use the UTF-8 encoded version.

Thanks,

Jeff

RE: Kodi Selective Cleaner addon support - gibxxi - 2023-03-26

Hi Jeff,

Sorry m8, but you may not have seen my edit to that post above. I'm still getting an error. In the edit above, the subsequent lines i'm going to post below weren't part of the log as I'd not pressed the 'OK' button on the error dialog the 4 times it comes up, [at the point I checked] so these entries weren't all in the log at that point...

Code:
2023-03-25 22:23:14.459 T:24872   error <general>: EXCEPTION IN (D:\Kodi Media Center\portable_data\addons\plugin.program.kscleaner\resources\lib\common.py, LINE 99 "dbname = dbname.txt + dbver"): 'xml.etree.ElementTree.Element' object has no attribute 'txt'

2023-03-25 22:23:14.459 T:24872    info <general>: There is a problem with your MySQL configuration

2023-03-25 22:23:14.467 T:20456    info <general>: Loading skin file: DialogConfirm.xml, load type: KEEP_IN_MEMORY

2023-03-25 22:23:16.651 T:24872   error <general>: EXCEPTION IN (D:\Kodi Media Center\portable_data\addons\plugin.program.kscleaner\resources\lib\common.py, LINE 169 "db = mysql.connector.connect(**config)"): connect() argument after ** must be a mapping, not NoneType

2023-03-25 22:23:16.651 T:24872    info <general>: There is a problem with your MySQL configuration

2023-03-25 22:23:17.884 T:24872   error <general>: EXCEPTION IN (D:\Kodi Media Center\portable_data\addons\plugin.program.kscleaner\resources\lib\common.py, LINE 123 "dbname = dbname.txt + dbver"): 'xml.etree.ElementTree.Element' object has no attribute 'txt'

2023-03-25 22:23:17.884 T:24872    info <general>: There is a problem with your MySQL configuration

2023-03-25 22:23:18.984 T:24872   error <general>: EXCEPTION IN (D:\Kodi Media Center\portable_data\addons\plugin.program.kscleaner\resources\lib\common.py, LINE 222 "db = mysql.connector.connect(**config)"): connect() argument after ** must be a mapping, not NoneType

2023-03-25 22:23:18.984 T:24872    info <general>: There is a problem with your MySQL configuration

Just to clarify, I downloaded the latest zip from Github. I can see you've made an edit 2 hours ago (at the time of writing). I removed the previous addon from Kodi completely, thinking it may not update with the downloaded one because they share the same version number. Exited Kodi, restarted, then installed from zip file with the new downloaded file, so I can be sure I'm using the latest fix. The above is what appears in the log when launching the addon.

Dan / Gib.

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-26

(2023-03-26, 00:28)gibxxi Wrote: Hi Jeff,

Sorry m8, but you may not have seen my edit to that post above. I'm still getting an error. In the edit above, the subsequent lines i'm going to post below weren't part of the log as I'd not pressed the 'OK' button on the error dialog the 4 times it comes up, [at the point I checked] so these entries weren't all in the log at that point...

Code:
2023-03-25 22:23:14.459 T:24872 error <general>: EXCEPTION IN (D:\Kodi Media Center\portable_data\addons\plugin.program.kscleaner\resources\lib\common.py, LINE 99 "dbname = dbname.txt + dbver"): 'xml.etree.ElementTree.Element' object has no attribute 'txt' 2023-03-25 22:23:14.459 T:24872 info <general>: There is a problem with your MySQL configuration 2023-03-25 22:23:14.467 T:20456 info <general>: Loading skin file: DialogConfirm.xml, load type: KEEP_IN_MEMORY 2023-03-25 22:23:16.651 T:24872 error <general>: EXCEPTION IN (D:\Kodi Media Center\portable_data\addons\plugin.program.kscleaner\resources\lib\common.py, LINE 169 "db = mysql.connector.connect(**config)"): connect() argument after ** must be a mapping, not NoneType 2023-03-25 22:23:16.651 T:24872 info <general>: There is a problem with your MySQL configuration 2023-03-25 22:23:17.884 T:24872 error <general>: EXCEPTION IN (D:\Kodi Media Center\portable_data\addons\plugin.program.kscleaner\resources\lib\common.py, LINE 123 "dbname = dbname.txt + dbver"): 'xml.etree.ElementTree.Element' object has no attribute 'txt' 2023-03-25 22:23:17.884 T:24872 info <general>: There is a problem with your MySQL configuration 2023-03-25 22:23:18.984 T:24872 error <general>: EXCEPTION IN (D:\Kodi Media Center\portable_data\addons\plugin.program.kscleaner\resources\lib\common.py, LINE 222 "db = mysql.connector.connect(**config)"): connect() argument after ** must be a mapping, not NoneType 2023-03-25 22:23:18.984 T:24872 info <general>: There is a problem with your MySQL configuration

Just to clarify, I downloaded the latest zip from Github. I can see you've made an edit 2 hours ago (at the time of writing). I removed the previous addon from Kodi completely, thinking it may not update with the downloaded one because they share the same version number. Exited Kodi, restarted, then installed from zip file with the new downloaded file, so I can be sure I'm using the latest fix. The above is what appears in the log when launching the addon.

Dan / Gib.

Ok, this is a different error, due to you using named databases vs. the default naming. I suspected I might run into this. You are the first to use named databases with the addon. It should be a simple fix. I'll need a debug for this and to know the exact names of your music and video databases in MySQL.

Thanks,

Jeff

RE: Kodi Selective Cleaner addon support - gibxxi - 2023-03-26

The names of the databases are exactly named as they were defined in the '<name>' tags in the advancedsettings.xml files I PM'd you, with the database 'version' number tacked on the end. So for Nexus:

Code:
<name>kodi_video_v</name>

becomes:

Code:
kodi_video_v121

on the MySQL (MariaDB 10) server.

Code:
<name>kodi_music_v</name>

becomes:

Code:
kodi_music_v82

on the MySQL (MariaDB 10) server.

As was previously alluded to earlier in this thread, Kodi automatically tacks the database version number onto the end of the specified 'name' of the database at database creation time. I need this feature as I have several databases running on the server, and I don't want to get confused and accidentally do something like drop the wrong database, or something.

Dan / Gib.

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-26

(2023-03-26, 00:28)gibxxi Wrote: Just to clarify, I downloaded the latest zip from Github. I can see you've made an edit 2 hours ago (at the time of writing). I removed the previous addon from Kodi completely, thinking it may not update with the downloaded one because they share the same version number. Exited Kodi, restarted, then installed from zip file with the new downloaded file, so I can be sure I'm using the latest fix. The above is what appears in the log when launching the addon.

Actually I think I have it fixed. I updated 0.0.9i again and this hopefully fixes the named database issue.

Thanks,

jeff

RE: Kodi Selective Cleaner addon support - gibxxi - 2023-03-26

Hi Jeff,

BINGO! We have a Winner! lol.

Wink

It's working. Well done, we got there in the end. Thanks for all your help, now I can go have a play and see if I can right-royally screw up my databases even more than what they are now.

Big Grin

Dan / Gib.

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-26

(2023-03-26, 00:54)gibxxi Wrote: As was previously alluded to earlier in this thread, Kodi automatically tacks the database version number onto the end of the specified 'name' of the database at database creation time. I need this feature as I have several databases running on the server, and I don't want to get confused and accidentally do something like drop the wrong database, or something.

Yep. The addon checks for the Kodi version number and appends the database to the end, as you've seen. One of the Kodi developers indicated this is how ti works. Glad you are working now.

Thanks,

jeff

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-26

I have posted a new version of the Kodi Selective Cleaner, version 0.0.9j. I found a major bug in the actor table analyzer, which @HeresJohnny had originally identified. This bug would cause actors to be listed which didn't have entries in the actor_link table. The problem is that the actor table also needs to be compared to the director_link and writer_link tables to determine if the actor record is orphaned or not. This has now been fixed. Unfortunately if you've used prior versions of the addon and have cleaned your actor table then you may be missing artwork and links to some writers and directors (those who weren't also actors). I am so sorry for this miss but that is the danger of alpha code.

In this version I have added support for named databases in MySQL, fixed an XML parsing issue of the advancedsettings.xml file and improved some of the notification / logging messages when there is a problem with the advancedsettings.xml configuration or connecting to your MySQL databases. This version adds a setting to determine both music and video database types and supports running in mixed mode (i.e. MySQL and local simultaneously.)

I'd like to thank @HeresJohnny and @gibxxi for all of their help testing and feature suggestions.

At this point I am stopping feature development for a while and will do bug fixes so I can soon get this to a 1.0 release.

Thanks,

Jeff

RE: Kodi Selective Cleaner addon support - HeresJohnny - 2023-03-26

Some breakage is to be expected since it's clearly marked as alpha. I'm having a good time testing and suggesting, that is what counts.

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-27

(2023-03-26, 23:33)HeresJohnny Wrote: Some breakage is to be expected since it's clearly marked as alpha. I'm having a good time testing and suggesting, that is what counts.

I appreciate the understanding, suggestions and testing assistance. I just hope it is adding value and helping folks clean things up a bit.

Jeff

RE: Kodi Selective Cleaner addon support - Pr.Sinister - 2023-03-29

Hey Jeff!

Glad to see you are on the case with helping us clean our libraries. Although Mezzmo is not for me, i saw how well coded it was when you asked me to test something with my huge MySQL database.

I feel the built-in Kodi Cleaner is severely lacking in some respects and i have been testing your addon with great results. I was trying to clean stuff manually yesterday and stumbled upon this thread.

Here are some screenshots of my success Big Grin

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-29

(2023-03-29, 17:22)Pr.Sinister Wrote: Hey Jeff!

Glad to see you are on the case with helping us clean our libraries. Although Mezzmo is not for me, i saw how well coded it was when you asked me to test something with my huge MySQL database.

I feel the built-in Kodi Cleaner is severely lacking in some respects and i have been testing your addon with great results. I was trying to clean stuff manually yesterday and stumbled upon this thread.

Here are some screenshots of my success :

Thanks for the feedback and the test results. As you indicated, a lot of the Kodi Selective Cleaner code is based upon Mezzmo code and the reverse engineering of the Kodi video database. Since you tested Mezzmo previously I've completed syncing all Kodi tables in the video database to Mezzmo (except country, which Mezzmo doesn't support). There were enough Mezzmo users who liked the native Kodi skin interface that it made sense to complete, which gave the underpinnings for the cleaner addon.

I am preparing to take this cleaner addon to a 1.0 release code soon. Suggestions and feedback are welcome and you can give a big shout out to @HeresJohnny for nudging me to add MySQL support. He was pleasantly persistent. I enjoyed figuring out how to interface Kodi, Python and MySQL. Your screenshots above validate the need for the cleaner for Kodi local and MySQL users. I run it on my Kodi instance and it almost always tells me nothing needs cleaned Smile

I did find a bug in the Mezzmo addon code once by using the cleaner.

Out of curiosity how fast or how long did it take for the cleaning on those tables ? I would expect subsequent cleaning will not take nearly as long since it won't be building up over a long period of time.

Thanks,

Jeff

RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-29

(2023-03-25, 14:51)HeresJohnny Wrote: I would like to throw some ideas around what I would love to see checked in the music database. Really, just a brainstorming and some ideas might be outside the scope of this add-on.

1. Table "album artist"
This table ties an album artist to an artist ID and an album ID
a) Apparently due to some manual tagging errors on my part some album artists appear with their musicbrainz ID. It would be great if the Analyzer could tell me which artist and album this concerns. Of course, corrections need then to be applied at file tag level.
b) Theoretically there could be cases where the same album artist name is linked to different artist IDs. It would be great if the Analyzer could point out these occurences.

2. Table "artist"
Most importantly, this table ties a local artist name to an artist ID entry in the musicbrainz database.
Even though I have meticulously tagged my files there are literally thousands of entries without musicbrainz IDs in this table and I'm unsure why this is. I suspect that additional tags other than "album artist" and "artist" are evaluated for this table, like composer and performer.
My OCD compels me to want to consolidate matching names and I wonder if and how a music database analyzer could support me in this. A listing of matching names with and without musicbrainz IDs might be within the scope of this add-on. Ideally an (additional) add-on would be able to (semi-automatically after revision by the user) apply corrections and write musicbrainz ID to green-lighted entries.

3. Table "role"
Don't even get me started... in my case there are over thousand entries, many containing names instead of functions or instruments... I suspect this is a consequence of a mixture of bad tags and bad musicbrainz data quality. I'm open for suggestions but I'm not optimistic to be able to fix this... ever :-)

I've started looking at the music database schema. The good news is that there are far less linked tables than the video database. The bad news is that there are many more instances for data integrity opportunity errors, like you are pointing out above with the musicbrainz tags. We'll have to have more discussions on how to represent data integrity issues vs. linked table errors. Linked table errors are easy and what the video analyzer focuses on mostly. It has a few basic integrity checking features but mainly when an item is in a linked table but not in the table entry for the element itself (i.e. genre missing in the movie table but exists in genre / genre_link)

For example on your first ask, it would be easy to run a query to see which album artists have a missing musicbrainz field. The more difficult piece would be to determine whether it is the correct value when it does exist. Right now there are no look-up or linked tables for the musicbrainz data in the Kodi music database. It all comes from the tags in the files. But I think this integrity piece could be handled by rescanning in your files and letting Kodi handle the missing / incorrect musicbtrainz info (assuming the files are tagged right).

For #3 the easy part would be missing roles or links to roles which don't exist. For the data integrity piece, the challenge would be understanding a list of valid roles (i.e. like having a musicbrainz look-up table). With that I could easily identify instances where roles are invalid (i.e. don't match the approved list of roles).

All of this starts to feel like a musicbrainz interface / look-up may be needed. I poked around on their website a bit and it doesn't look extremely difficult but a full clean could pound their servers, which they may not appreciate. Overall though I am still a bit at a loss on what the musicbrainz info is really used for inside Kodi. I do see some value in the musicbrainz id being used to differentiate two albums or artists with the same name for album or artwork alignment but beyond that it is less clear..

I am still poking around the music database.

Thanks,

Jeff

RE: Kodi Selective Cleaner addon support - HeresJohnny - 2023-03-29

I think it would be a really good idea not (only) to throw around ideas between us but for you to have a trialogue with the two main developers of the music database outside this forum. It is their brain child and they will have already identified - or at least discussed - many things that would need adressing. I don't know how available they are, though.

Of course, I would be pleased to help from a user perspective.

I have also played around some more with the video database analyzer. Especially when it comes to tv shows I hadn't noticed before that even there was nothing left to clean, the analyzer would still point out lots of errors, mainly ratings mismatches and episode ID mismatches. The latter took quite some time to fix as I first had to go and identify related shows via the database for those mismatches, then re-scrape them and finally run ratings add-ons on them. In a far away future world, your add-on might offer additional functionality to take the user to the tv show information screen for re-scraping to correct such mismatches less painfully.

I was amazed by how many ID mismatches existed even for shows/episodes that I had only scraped this year. That is probably due to scraper changes which will forever be adapting to changing source websites / APIs. I feel those were pretty frequent in the past year with the TVDB especially, so maybe that's why.

On the performance side, the add-on is pretty fast now. After correcting all mismatches through rescraping the analyzer cleaned out hundreds of season mismatches by itself which took around 3-minutes on my oldish "server". However, one should avoid clicking in Kodi too much as the interface appears to freeze in between.

Finally, just to remind everyone, a Kodi MySQL setup is still considered experimental by the developers and everybody who goes down that road should be able to help themselves instead of asking for support unitl such a time as that setup is supported natively in more user friendly ways. Now, though, we have a great tool in the form of your add-on that helps with troubleshooting.