+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Add-on Support (https://forum.kodi.tv/forumdisplay.php?fid=27)
+---- Forum: Program Add-ons (https://forum.kodi.tv/forumdisplay.php?fid=151)
+---- Thread: Release Kodi Selective Cleaner addon support (/showthread.php?tid=369984)
RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-09
(2023-03-09, 04:22)HeresJohnny Wrote: I've tested it on a portable installation of Nexus and a nightly of Omega. On Nexus it connects to the databases and returns counts of objects, on Omega I get error messages of "Database not found". Also, the add-on installs to plugin.program.kcsleaner which I'm guessing should be plugin.program.kscleaner :-)
Thanks for confirming it works on Nexus and catching my typo in the addon name. I have fixed it for the next test release. I haven't added support for Omega yet. I haven't setup a test environment for it and the database numbers aren't posted yet on the Kodi Wiki. If you can tell me the database names for the video and music databases (i.e. myvideos122 and mymusic83 etc..) I can add support for Omega. I've made more progress and have the movies and TV Shows browser working with MySQL, as well as the CSV export utility. The analyzer module is going to take some time due to the sheer number of SQL queries involved and I don't plan to make the database backup work with MySQL. If someone is running MySQL, they will need to have enough DBA skills to handle their own backups. I'll likely posted an update early next week which should have feature parity between SQLite and MySQL for everything except the analyzer module and database backups.
I am already seeing something interesting where the Kodi scrapers are adding a season -1 and 0 to all TV Shows. I am not sure why this is but am sure there is a purpose. When I sync Kodi data from Mezzmo I don't add these seasons to the Kodi database (except when there is an episode with a season 0). I'll need to account for this in the TV Show portion of the analyzer because it will toss out a mismatch where there is a season entry in the seasons table with no associated episodes. This will likely be a Team Kodi question as to the purpose of seasons -1 and 0..
Thanks,
Jeff
RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-10
(2023-03-09, 11:33)jbinkley60 Wrote: Thanks for confirming it works on Nexus and catching my typo in the addon name. I have fixed it for the next test release. I haven't added support for Omega yet. I haven't setup a test environment for it and the database numbers aren't posted yet on the Kodi Wiki. If you can tell me the database names for the video and music databases (i.e. myvideos122 and mymusic83 etc..) I can add support for Omega. I've made more progress and have the movies and TV Shows browser working with MySQL, as well as the CSV export utility. The analyzer module is going to take some time due to the sheer number of SQL queries involved and I don't plan to make the database backup work with MySQL. If someone is running MySQL, they will need to have enough DBA skills to handle their own backups. I'll likely posted an update early next week which should have feature parity between SQLite and MySQL for everything except the analyzer module and database backups.
I am already seeing something interesting where the Kodi scrapers are adding a season -1 and 0 to all TV Shows. I am not sure why this is but am sure there is a purpose. When I sync Kodi data from Mezzmo I don't add these seasons to the Kodi database (except when there is an episode with a season 0). I'll need to account for this in the TV Show portion of the analyzer because it will toss out a mismatch where there is a season entry in the seasons table with no associated episodes. This will likely be a Team Kodi question as to the purpose of seasons -1 and 0..
@HeresJohnny I've uploaded a new test release that fixes the addon naming, adds support for Omega and MySQL should now work for all menu options except the video analyzer, music analyzer and backups. Note that the top 4 menu options (Movies, TV Shows, Music Videos and Music) just allow you to browse your library hierarchy. I had originally planned to have them be able to delete individual files or a whole set of files. I've never added that functionality yet. I felt it would be a more friendly way of folks being able to delete library items and ensure all artifacts are removed. if there is ever interest I will complete that coding.
I am traveling for the next few days. Next week I'll begin working on the video analyzer module.
Thanks,
Jeff
RE: Kodi Selective Cleaner addon support - HeresJohnny - 2023-03-12
Hi Jeff,
the results of browsing the libraries are inconsisten. For example, in movies I see entries for 1,2,3,4,5,9 and the letters. Browsing 1,2 and 9 throws this error:
Browsing the TV shows immediately throws the same error:xml:
2023-03-12 14:54:34.490 T:15872 error <general>: KS Cleaner Movies error.
2023-03-12 14:54:34.512 T:15872 error <general>: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
- NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
Error Type: <class 'UnboundLocalError'>
Error Contents: local variable 'curpf' referenced before assignment
Traceback (most recent call last):
File "C:\Users\JPS\AppData\Roaming\Kodi\addons\plugin.program.kscleaner\resources\lib\movies.py", line 78, in displayMovies
if len(movie[2]) < 1: # Handle blank movie names
TypeError: object of type 'int' has no len()
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\JPS\AppData\Roaming\Kodi\addons\plugin.program.kscleaner\default.py", line 2, in <module>
from resources.lib import menu
File "C:\Users\JPS\AppData\Roaming\Kodi\addons\plugin.program.kscleaner\resources\lib\menu.py", line 151, in <module>
displayMenu() # Display main menu
File "C:\Users\JPS\AppData\Roaming\Kodi\addons\plugin.program.kscleaner\resources\lib\menu.py", line 127, in displayMenu
displayMovieMenu(dbtype)
File "C:\Users\JPS\AppData\Roaming\Kodi\addons\plugin.program.kscleaner\resources\lib\movies.py", line 54, in displayMovieMenu
displayMovies(kmmovies[vdate][0], dbtype)
File "C:\Users\JPS\AppData\Roaming\Kodi\addons\plugin.program.kscleaner\resources\lib\movies.py", line 88, in displayMovies
del curpf
UnboundLocalError: local variable 'curpf' referenced before assignment
-->End of Python script error report<--
All other libraries and entries browse correctly.python:
2023-03-12 14:57:52.975 T:15900 error <general>: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
- NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
Error Type: <class 'UnboundLocalError'>
Error Contents: local variable 'curpf' referenced before assignment
Traceback (most recent call last):
File "C:\Users\JPS\AppData\Roaming\Kodi\addons\plugin.program.kscleaner\resources\lib\tvshows.py", line 33, in displayTvshows
if len(tvshow[1]) < 1: # Handle blank TV Show names
TypeError: object of type 'int' has no len()
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\JPS\AppData\Roaming\Kodi\addons\plugin.program.kscleaner\default.py", line 2, in <module>
from resources.lib import menu
File "C:\Users\JPS\AppData\Roaming\Kodi\addons\plugin.program.kscleaner\resources\lib\menu.py", line 151, in <module>
displayMenu() # Display main menu
File "C:\Users\JPS\AppData\Roaming\Kodi\addons\plugin.program.kscleaner\resources\lib\menu.py", line 130, in displayMenu
displayTvshows(dbtype)
File "C:\Users\JPS\AppData\Roaming\Kodi\addons\plugin.program.kscleaner\resources\lib\tvshows.py", line 44, in displayTvshows
del curpf
UnboundLocalError: local variable 'curpf' referenced before assignment
-->End of Python script error report<--
RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-12
(2023-03-12, 15:59)HeresJohnny Wrote: All other libraries and entries browse correctly.
Thanks for the catch on the extra del curpf commands. Those are for SQLite not MySQL. I coded the movies and TV Shows MySQL changes first and never went back and fixed this when I caught it later. I've uploaded a fix, if you could try it. Did you try any CSV exports of your MySQL tables ? I'd like to ensure that works for you too. I plan to have the first part of the analyzer function working in the next couple of days with MySQL. I'll do actor and actor_link first. Once those are confirmed working I can go through the rest of the tables since they are pretty much use the same types of queries just with different table variables to analyze. It's just slow tedious work so it will take a little time.
Thanks,
Jeff
RE: Kodi Selective Cleaner addon support - HeresJohnny - 2023-03-12
With the update, the movies that threw errors before now throw a general error on the main screen: "Error connecting to the Kodi databases".
CSV seem to export just fine, although personally I wouldn't have a use case for them just yet.
RE: Kodi Selective Cleaner addon support - HeresJohnny - 2023-03-12
Btw, as I thought, MySQL syntax would differ in various places from SQLite syntax. Maybe you could document your findings in a table and put in the Kodi Wiki ? I know it's asking a lot of your free time and I don't know if a comprehensive side-by-side comparison is even possible but that would certainly help other developers broaden their horizons for inclusion of MySQL in their add-ons.
RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-12
(2023-03-12, 22:16)HeresJohnny Wrote: With the update, the movies that threw errors before now throw a general error on the main screen: "Error connecting to the Kodi databases".
CSV seem to export just fine, although personally I wouldn't have a use case for them just yet.
Ok. I may have a mix of code versions posted right now due to all the changes. Let me finish the actor and actor_link analyzer sections and I will retest prior to posting. I have the actor section done except for the clean function but it looks like a simple change. If so, I should be able to complete actor_link tomorrow or Tuesday and post it for testing.
Thanks,
Jeff
RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-12
(2023-03-12, 22:22)HeresJohnny Wrote: Btw, as I thought, MySQL syntax would differ in various places from SQLite syntax. Maybe you could document your findings in a table and put in the Kodi Wiki ? I know it's asking a lot of your free time and I don't know if a comprehensive side-by-side comparison is even possible but that would certainly help other developers broaden their horizons for inclusion of MySQL in their add-ons.
I'll give it some thought. Much of this is on the Internet already. Most folks don't read / write directly to Kodi's database so it may have nominal to moderate value. This exercise has given me much more appreciation to what folks go through to maintain their Kodi data integrity and makes me really appreciate my Mezzmo setup.
Thanks,
Jeff
RE: Kodi Selective Cleaner addon support - HeresJohnny - 2023-03-13
I had installed your Mezzmo add-on before as it promised to keep Kodi databases in sync and eliminate errors on the way. However, that was then only true of SQLite local databases as I realized. If it were possible to be error-free via Mezzmo in a MySQL setup as a result of your work here I would certainly consider going that route.
RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-13
(2023-03-13, 00:34)HeresJohnny Wrote: I had installed your Mezzmo add-on before as it promised to keep Kodi databases in sync and eliminate errors on the way. However, that was then only true of SQLite local databases as I realized. If it was possible to be error-free via Mezzmo in a MySQL setup as a result of your work here I would certainly consider going that route.
Understood. Mezzmo isn't an adjunct to MySQL, it is an alternative. I don't see a situation where I would extend the Mezzmo Kodi addon to write to a MySQL environment. The whole purpose of this Kodi Selective Cleaner addon is to make it easier for folks like you who run native and MySQL databases. I myself will rarely use it but it should be of value to others. FYI, I got the actor table analyzer working with MySQL. The actor_link analyzer should be ready tomorrow and I will post a complete update. What made me appreciate this more was that in my simple testing of adding and deleting content to my MySQL Kodi instance I generated 120 orphaned actor records very quickly. I doubt most folks would notice since actor_link is more important for many Kodi operations. I'll be curious if I generated any orphaned actor_link table entries.
Thanks,
Jeff
RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-13
(2023-03-12, 22:16)HeresJohnny Wrote: With the update, the movies that threw errors before now throw a general error on the main screen: "Error connecting to the Kodi databases".
CSV seem to export just fine, although personally I wouldn't have a use case for them just yet.
I have posted version 0..0.9c which has the Video Db Data Analyzer working now for the actor and actor_link tables for MySQL. All three options, Analyze, Analyze / CSV Export and Analyze / Clean Table should work. I highly recommend making a database backup before using the cleaner function. I've tested it on my test setup but backups are always good.
Thanks,
Jeff
RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-13
(2023-03-09, 11:33)jbinkley60 Wrote: I am already seeing something interesting where the Kodi scrapers are adding a season -1 and 0 to all TV Shows. I am not sure why this is but am sure there is a purpose. When I sync Kodi data from Mezzmo I don't add these seasons to the Kodi database (except when there is an episode with a season 0). I'll need to account for this in the TV Show portion of the analyzer because it will toss out a mismatch where there is a season entry in the seasons table with no associated episodes. This will likely be a Team Kodi question as to the purpose of seasons -1 and 0..
@Klojum @Karellen I am hoping either of you might have a quick answer to the purpose of seasons -1 and season 0 being added to the Kodi video database seasons table when a new TV Show is scraped ? Right now they show unmatched in the video database analyzer (i.e. they have no corresponding episodes). I tested scraping season 1 of a series. I didn't test scraping a higher season to see if Kodi creates all lower seasons (i.e. scrape season 3 and does Kodi creates -1, 0, 1 and 2) or is this something specific to seasons -1 and 0.
Thanks,
Jeff
RE: Kodi Selective Cleaner addon support - Klojum - 2023-03-13
Season 0 is for the TV show's specials (Christmas episodes or season features etc).
Season -1 ... Can't remember.
RE: Kodi Selective Cleaner addon support - Karellen - 2023-03-13
(2023-03-13, 15:46)jbinkley60 Wrote: purpose of seasons -1 and season 0 being added to the Kodi video database seasons table when a new TV Show is scraped ?I have no idea. I have never noticed it before.
RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-13
(2023-03-13, 20:01)Klojum Wrote: Season 0 is for the TV show's specials (Christmas episodes or season features etc).
Season -1 ... Can't remember.
Thanks. I knew that for season 0 but what was a bit of a surprise is that the season is being created with no associated episodes even if there are no specials, special features or season 0 items. I'll likely end up reporting on them but coding around cleaning those seasons unless they are the only 2 seasons and have no associated episodes.
Jeff