+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Add-on Support (https://forum.kodi.tv/forumdisplay.php?fid=27)
+---- Forum: Program Add-ons (https://forum.kodi.tv/forumdisplay.php?fid=151)
+---- Thread: Release Kodi Selective Cleaner addon support (/showthread.php?tid=369984)
RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-13
@HeresJohnny I ended up making great progress today on the Video DB Data Analyzer code. I posted version 0.0.9d which should work on all tables up through the musicvideo table with MySQL. All three options, Analyze, Analyze / CSV Export and Analyze / Clean Table should work. I highly recommend making a database backup before using the cleaner function. I've tested it on my test setup but backups are always good.
Thanks,
Jeff
RE: Kodi Selective Cleaner addon support - HeresJohnny - 2023-03-14
Great work! Ran it against my database... over 30.000 actor_link entries were cleaned, that took a good chunk of time. Maybe a progress bar would help show the user that Kodi is still alive :-)
RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-14
(2023-03-14, 02:08)HeresJohnny Wrote: Great work! Ran it against my database... over 30.000 actor_link entries were cleaned, that took a good chunk of time. Maybe a progress bar would help show the user that Kodi is still alive :-)
Thanks for the update. I'll give the progress bar some thought. I would suspect normally it will be much faster after an initial cleaning. You've helped prove my point about how messy the databases can become. I have the remainder of the tables done, I just need to finish some final testing. I added / modified a few hundred lines of code. I should post a version tomorrow with all tables in the menu working with MySQL.
Thanks,
Jeff
RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-14
(2023-03-14, 02:08)HeresJohnny Wrote: Great work! Ran it against my database... over 30.000 actor_link entries were cleaned, that took a good chunk of time. Maybe a progress bar would help show the user that Kodi is still alive :-)
I've posted version 0.0.9e which has support for all tables with MySQL. I looked at what it would take to add a progress dialog box during cleaning. It's a small amount of work so I'll look to implement in the next release. I also see that some of the slowness is due to me opening / closing a MySQL cursor on each clean action on every item. I'll see what it would take to reuse the cursor for all clean actions in a single run which should speed things up significantly. SQLite doesn't do this. It does them as a single batch.
Thanks,
Jeff
RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-14
(2023-03-14, 02:08)HeresJohnny Wrote: Great work! Ran it against my database... over 30.000 actor_link entries were cleaned, that took a good chunk of time. Maybe a progress bar would help show the user that Kodi is still alive :-)
I was able to punch out a shared MySQL cursor solution with version 0.0.9f . It should be much faster now cleaning large tables. Progress bar is for another day but I don't think it may be needed now.
Thanks,
Jeff
RE: Kodi Selective Cleaner addon support - HeresJohnny - 2023-03-14
All videodatabase tables cleaned out nicely with 0.0.9f. However, I triggered another error while browsing movies (obfuscating login data):
xml:2023-03-14 14:26:29.211 T:9624 debug <general>: Kodi selective cleaner movie menu selection is: 1
2023-03-14 14:26:29.211 T:9624 debug <general>: KC Cleaner Movies Menu selection: 1
2023-03-14 14:26:29.213 T:9624 debug <general>: KS Cleaner parse: mysql 127.0.0.1 3306 xxx xxx myvideos121
2023-03-14 14:26:29.295 T:9624 error <general>: KS Cleaner Movies error.
2023-03-14 14:26:29.296 T:9624 error <general>: EXCEPTION IN (C:\Kodi\portable_data\addons\plugin.program.kscleaner\resources\lib\movies.py, LINE 78 "if len(movie[2]) < 1: # Handle blank movie names"): object of type 'int' has no len()
Also, strangely enough, there seems to be a movie with a blank entry
xml:
2023-03-14 14:26:25.712 T:9624 debug <general>: Kodi selective cleaner movie menu selection is:
2023-03-14 14:26:25.712 T:9624 debug <general>: KC Cleaner Movies Menu selection:
2023-03-14 14:26:25.714 T:9624 debug <general>: KS Cleaner parse: mysql 127.0.0.1 3306 xxx xxx myvideos121
2023-03-14 14:26:25.879 T:9624 error <general>: KS Cleaner Movies error.
2023-03-14 14:26:25.879 T:9624 error <general>: EXCEPTION IN (C:\Kodi\portable_data\addons\plugin.program.kscleaner\resources\lib\movies.py, LINE 78 "if len(movie[2]) < 1: # Handle blank movie names"): object of type 'int' has no len()
As far as I know, that is "The Doom that came to Gotham" which was scraped with UMS. Would it be possible to log which line in the affected table triggered the error?
RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-14
(2023-03-14, 15:34)HeresJohnny Wrote: All videodatabase tables cleaned out nicely with 0.0.9f. However, I triggered another error while browsing movies (obfuscating login data):
xml:2023-03-14 14:26:29.211 T:9624 debug <general>: Kodi selective cleaner movie menu selection is: 1
2023-03-14 14:26:29.211 T:9624 debug <general>: KC Cleaner Movies Menu selection: 1
2023-03-14 14:26:29.213 T:9624 debug <general>: KS Cleaner parse: mysql 127.0.0.1 3306 xxx xxx myvideos121
2023-03-14 14:26:29.295 T:9624 error <general>: KS Cleaner Movies error.
2023-03-14 14:26:29.296 T:9624 error <general>: EXCEPTION IN (C:\Kodi\portable_data\addons\plugin.program.kscleaner\resources\lib\movies.py, LINE 78 "if len(movie[2]) < 1: # Handle blank movie names"): object of type 'int' has no len()
Also, strangely enough, there seems to be a movie with a blank entry
xml:
2023-03-14 14:26:25.712 T:9624 debug <general>: Kodi selective cleaner movie menu selection is:
2023-03-14 14:26:25.712 T:9624 debug <general>: KC Cleaner Movies Menu selection:
2023-03-14 14:26:25.714 T:9624 debug <general>: KS Cleaner parse: mysql 127.0.0.1 3306 xxx xxx myvideos121
2023-03-14 14:26:25.879 T:9624 error <general>: KS Cleaner Movies error.
2023-03-14 14:26:25.879 T:9624 error <general>: EXCEPTION IN (C:\Kodi\portable_data\addons\plugin.program.kscleaner\resources\lib\movies.py, LINE 78 "if len(movie[2]) < 1: # Handle blank movie names"): object of type 'int' has no len()
As far as I know, that is "The Doom that came to Gotham" which was scraped with UMS. Would it be possible to log which line in the affected table triggered the error?
Sure. I can also check for blanks better to avoid these exceptions. You should be able to see the specifics by doing a CSV export of the movie table.
Thanks,
Jeff
RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-14
(2023-03-14, 15:34)HeresJohnny Wrote: As far as I know, that is "The Doom that came to Gotham" which was scraped with UMS. Would it be possible to log which line in the affected table triggered the error?
Do me a favor and don't fix the error in your database. let me code a fix and then we can test it. I'll probably write a message to the logs so you can see the error message and maybe a popup dialog box to for immediate visibility.
Jeff
RE: Kodi Selective Cleaner addon support - HeresJohnny - 2023-03-14
(2023-03-14, 15:34)HeresJohnny Wrote: Also, strangely enough, there seems to be a movie with a blank entryTurns out, I forgot to change "Get title" from IMDB to themoviedb for one source...
RE: Kodi Selective Cleaner addon support - HeresJohnny - 2023-03-14
(2023-03-14, 15:44)jbinkley60 Wrote: Do me a favor and don't fix the error in your database. let me code a fix and then we can test it. I'll probably write a message to the logs so you can see the error message and maybe a popup dialog box to for immediate visibility.
Sorry, already fixed it. I can reproduce it again by emptying the culprit field in the db, though.
RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-14
(2023-03-14, 15:58)HeresJohnny Wrote:(2023-03-14, 15:34)HeresJohnny Wrote: Also, strangely enough, there seems to be a movie with a blank entryTurns out, I forgot to change "Get title" from IMDB to themoviedb for one source...
So it sounds like the tool is handy at helping keep things cleaner. I presume the cleaning process was much faster with the shared cursor ?
Jeff
RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-14
(2023-03-14, 16:00)HeresJohnny Wrote:(2023-03-14, 15:44)jbinkley60 Wrote: Do me a favor and don't fix the error in your database. let me code a fix and then we can test it. I'll probably write a message to the logs so you can see the error message and maybe a popup dialog box to for immediate visibility.
Sorry, already fixed it. I can reproduce it again by emptying the culprit field in the db, though.
If you'd like to try again. I added a check for a blank title field. Version 0.0.9g has the update. It will show in the listing that the title is invalid and write it to the addon logs.
Jeff
RE: Kodi Selective Cleaner addon support - HeresJohnny - 2023-03-15
Hm, I'm getting "Can't connect to Kodi databases" again for Movies and TV shows.
RE: Kodi Selective Cleaner addon support - jbinkley60 - 2023-03-15
(2023-03-15, 23:02)HeresJohnny Wrote: Hm, I'm getting "Can't connect to Kodi databases" again for Movies and TV shows.
That is odd. Can you posted the exception message out of the kodi.log file so I an see what is happening ? I double checked that I uploaded the correct code and it is working for me. also please make sure it says version 0.0.9g and you didn't accidentally load an older version.
Jeff
RE: Kodi Selective Cleaner addon support - HeresJohnny - 2023-03-16
The error message is the same as in post #82. I'm guessing it's either the movie title length or a special character that triggers the error in the case of all TV shows and Movies that start with 1,2 or 9. For example, there's a TV show with the title
Quote:'Allo 'Allo!and movies with titles like
Quote:1900
2012
23
9
If you want, I can get you access to my server for testing or provide you with a database dump - please PM me for that.