+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Add-on Support (https://forum.kodi.tv/forumdisplay.php?fid=27)
+---- Forum: Program Add-ons (https://forum.kodi.tv/forumdisplay.php?fid=151)
+---- Thread: [RELEASE] - Thumbnails Cleaner v1.1.4 (/showthread.php?tid=158972)
RE: [RELEASE] - Thumbnails Cleaner - User 34959 - 2013-03-14
It is <name>xbmc_video</name> + the actual db version read from the version table. So with this config it's xbmc_video75. I'm fairly certain most of the mysql db users follow this same naming scheme and the advanced settings.xml looks the same.
If <name>blaablaa</name> is not used it's MyVideos75, even in mysql.
RE: [RELEASE] - Thumbnails Cleaner - m4x1m - 2013-03-14
(2013-03-14, 21:54)toiva Wrote: It is <name>xbmc_video</name> + the actual db version read from the version table. So with this config it's xbmc_video75. I'm fairly certain most of the mysql db users follow this same naming scheme and the advanced settings.xml looks the same.
If <name>blaablaa</name> is not used it's MyVideos75, even in mysql.
Ok dbutils.py modified for reflect this situation, please replace with this one
RE: [RELEASE] - Thumbnails Cleaner - User 34959 - 2013-03-14
Now it gets stuck at finalizing phase when running the simulation.
Code:
22:59:09 T:4632645632 ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
- NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
Error Type: <type 'exceptions.OSError'>
Error Contents: (1, 'Operation not permitted', '/Users/anssi/Pictures/Backdrops')
Traceback (most recent call last):
File "/Users/anssi/Library/Application Support/XBMC/addons/script.thumbnailscleaner/resources/lib/gui.py", line 155, in onClick
elif controlId == self.idSimulate: Cleaner( self ).doClean(True)
File "/Users/anssi/Library/Application Support/XBMC/addons/script.thumbnailscleaner/resources/lib/cleaner.py", line 301, in doClean
self.finalizeClean( doSimulate )
File "/Users/anssi/Library/Application Support/XBMC/addons/script.thumbnailscleaner/resources/lib/cleaner.py", line 334, in finalizeClean
for f in getFiles: os.remove(f)
OSError: (1, 'Operation not permitted', '/Users/anssi/Pictures/Backdrops')
-->End of Python script error report<--I've set /Users/anssi/Pictures/ as a target, is it trying to recursively delete the child directories? I also tried /Users/anssi/Pictures/test but it still fails with the same error.
Second issue, it does not seem to detect my music video posters. If you use Concertdb scraper it scrapes standard posters from TMDB. So music videos can use either poster or a thumb, depending on a scraper used.
Thanks for the efforts so far, we shall continue tomorrow.
RE: [RELEASE] - Thumbnails Cleaner - m4x1m - 2013-03-15
(2013-03-14, 22:56)toiva Wrote: I've set /Users/anssi/Pictures/ as a target, is it trying to recursively delete the child directories? I also tried /Users/anssi/Pictures/test but it still fails with the same error.
Second issue, it does not seem to detect my music video posters. If you use Concertdb scraper it scrapes standard posters from TMDB. So music videos can use either poster or a thumb, depending on a scraper used.
Thanks for the efforts so far, we shall continue tomorrow.
I changed the method to move files. Now will be deleted only the files that already exist in the destination folder, instead to entirely empty the folder. More safely. (Python: shutil.copy2 overwrites the files, shutil.move no
)Thanks for your time!
Download
script.thumbnailscleaner-1.0.4.zip
Code:
v1.0.4:
- Fixed all problems related to mysql database (for real).
- Added check for music video posters.
- Fixed issue when moving files in destination folder that already exist.RE: [RELEASE] - Thumbnails Cleaner - jmarshall - 2013-03-15
Nice work. May I suggest some changes:
1. Don't bother looking in MyVideos75.db. There's no need to do so. If you MUST look in there, then just use JSON-RPC to grab what you need - that way you're immune to db layout changes.
2. Do bother looking in Textures13.db. It's a local sqlite database only. The layout can and will change, but you can't do anything about that for now, so don't bother
All you need do is compare what's there with what's on disk.3. Further, you can then start doing cleanup based on the time and usage information stored in that database. e.g. you could clean up stuff that hasn't been used for ages. Note that this gets around having to play in the videodb or musicdb's - stale entries in Textures13.db gets cleaned out automatically with time.
4. Note that XBMC operates under the assumption that Textures.db is king - i.e. it doesn't matter if it's on disk, if it's not in the database it'll be recached. Thus, to recache stuff, just remove the entry in Textures.db (assuming ofc that the original URL is still valid). Further, this implies that anything on disk and not in Textures.db can be removed, and anything not on disk but in Textures.db is a stale entry in Textures.db so should be removed.
Lastly, once you've come up with a good system, we'll look to expose whatever you need via JSON-RPC so you're then immune to db-layout issues and no longer need a database connection. I suspect all you really need is a way to grab everything from the texture (joined size) tables?
Cheers,
Jonathan
RE: [RELEASE] - Thumbnails Cleaner - m4x1m - 2013-03-15
(2013-03-15, 03:06)jmarshall Wrote: 1. Don't bother looking in MyVideos75.db. There's no need to do so. If you MUST look in there, then just use JSON-RPC to grab what you need - that way you're immune to db layout changes.
From MyVideos75.db (MyMusic32) i get the the info from the art table, the script generates the hash and then excludes the images from the list of files to be deleted. Yes, I can convert the query to JSON, so I'll study to do this. (It's possible to get the content of the art table with JSON? I couldn't even get the version number in my previous attempt.)
(2013-03-15, 03:06)jmarshall Wrote: 2. Do bother looking in Textures13.db. It's a local sqlite database only. The layout can and will change, but you can't do anything about that for now, so don't bother
Yes, in fact I do this, exactly as MyVideos and MyMusic.
(2013-03-15, 03:06)jmarshall Wrote: 3. Further, you can then start doing cleanup based on the time and usage information stored in that database. e.g. you could clean up stuff that hasn't been used for ages. Note that this gets around having to play in the videodb or musicdb's - stale entries in Textures13.db gets cleaned out automatically with time.
In the settings I added a slider to set the days. I compare the days with the lasthashcheck field, then deletes the old. I added an option to delete the fields that have an empty lasthashcheck (I noticed that normally have an http:// url)
(2013-03-15, 03:06)jmarshall Wrote: 4. Note that XBMC operates under the assumption that Textures.db is king - i.e. it doesn't matter if it's on disk, if it's not in the database it'll be recached. Thus, to recache stuff, just remove the entry in Textures.db (assuming ofc that the original URL is still valid). Further, this implies that anything on disk and not in Textures.db can be removed, and anything not on disk but in Textures.db is a stale entry in Textures.db so should be removed.
Yes, I know this.
(2013-03-15, 03:06)jmarshall Wrote: Lastly, once you've come up with a good system, we'll look to expose whatever you need via JSON-RPC so you're then immune to db-layout issues and no longer need a database connection. I suspect all you really need is a way to grab everything from the texture (joined size) tables?
I'll work to do this!
In conclusion. I read all files present in the Thumbnails folder. I get arts info from the MyVideos db, calculates hash and excludes the files. Same thing with the MyMusic db. Process the textures table, deletes all fields with empty lasthashcheck (option), deletes all fields that are older than X days (optional), then compares the fileds that are remained with my list of files excluding the files that are present, and lastly copy/move the remained files, that can be safely to deleted. I think this is the correct way, but I could be wrong.
Thanks!
UPDATE: OK, I get the arts with JSON, except for the actors.
1. How to I get the actors thumbs?
2. The season-all-poster is missing when query GetSeasons.
2. Can I call the textures13.db with JSON?
RE: [RELEASE] - Thumbnails Cleaner - Milhouse - 2013-03-18
(2013-03-15, 12:31)m4x1m Wrote: UPDATE: OK, I get the arts with JSON, except for the actors.
1. How to I get the actors thumbs?
2. The season-all-poster is missing when query GetSeasons.
3. Can I call the textures13.db with JSON?
1) By adding the "cast" property to the VideoLibrary.GetMovies, VideoLibrary.GetTVShows and VideoLibrary.GetExpisodes methods. However JSON is a bit flaky with respect to cast thumbs as it will only return cast thumbs for manually scraped movies/tvshows, and won't return cast thumbs that are cached dynamically. There's a disconnect somewhere, probably highlighted by enabling "Download actor thumbnails" after performing a full scrape (you probably wouldn't notice the problem while scraping with it enabled).
2) Yes, probably because there is no concept of a "season all" in the database, only Season #0 (Special) through to Season #n. Unless there is a change forthcoming, you'll probably need to hack in season-all support. I suppose JSON could implement a "season-all" season somehow, perhaps using seasonid -1 (ugh), though this seasonid when calling GetEpisodes would not be valid so you'd need to special case it.
3) No, you need to use the sqlite3 module.
RE: [RELEASE] - Thumbnails Cleaner - m4x1m - 2013-03-18
(2013-03-18, 10:39)MilhouseVH Wrote:(2013-03-15, 12:31)m4x1m Wrote: UPDATE: OK, I get the arts with JSON, except for the actors.
1. How to I get the actors thumbs?
2. The season-all-poster is missing when query GetSeasons.
3. Can I call the textures13.db with JSON?
1) By adding the "cast" property to the VideoLibrary.GetMovies, VideoLibrary.GetTVShows and VideoLibrary.GetExpisodes methods. However JSON is a bit flaky with respect to cast thumbs as it will only return cast thumbs for manually scraped movies/tvshows, and won't return cast thumbs that are cached dynamically. There's a disconnect somewhere, probably highlighted by enabling "Download actor thumbnails" after performing a full scrape (you probably wouldn't notice the problem while scraping with it enabled).
2) Yes, probably because there is no concept of a "season all" in the database, only Season #0 (Special) through to Season #n. Unless there is a change forthcoming, you'll probably need to hack in season-all support. I suppose JSON could implement a "season-all" season somehow, perhaps using seasonid -1 (ugh), though this seasonid when calling GetEpisodes would not be valid so you'd need to special case it.
3) No, you need to use the sqlite3 module.
Thanks for the reply!
I spent long time testing json, but definitely, for me, in this case is better to use a raw approach to the database. To get season arts or episode thumbs is necessary, with json, create a double loop that slows the operation. With a raw approach I can get all info that I needed, directly, also the actors thumbs. Never mind if I have to keep up with the changes in the database layout.
RE: [RELEASE] - Thumbnails Cleaner - patria o muerte - 2013-03-18
What a addon!
I saved about 119 mb of disk space.
Very useful.
Thanks!
RE: [RELEASE] - Thumbnails Cleaner - DaDuck - 2013-03-18
Is there any way to set the cleaner to remove all the episodes thumbs?
RE: [RELEASE] - Thumbnails Cleaner - m4x1m - 2013-03-18
(2013-03-18, 12:14)patria o muerte Wrote: What a addon!
I saved about 119 mb of disk space.
Very useful.
Thanks!
(2013-03-18, 13:01)DaDuck Wrote: Is there any way to set the cleaner to remove all the episodes thumbs?
At the moment no, but in the future I would to add more options, but first I want to pick up some tips and be sure that everything works well!
RE: [RELEASE] - Thumbnails Cleaner - DaDuck - 2013-03-18
Okay
I have found a bug (?) anyway if you have this line in advanced settings the script don't work
<hideemptyseries>true</hideemptyseries>
RE: [RELEASE] - Thumbnails Cleaner - m4x1m - 2013-03-18
(2013-03-18, 15:38)DaDuck Wrote: Okay
I have found a bug (?) anyway if you have this line in advanced settings the script don't work
<hideemptyseries>true</hideemptyseries>
If in advancedsettings.xml there is an extraneous char like # (used normally as comment) the script fails because this char isn't part of xml format, for comment in xml uses <!-- -->. I think your problem is this, but if you want, you can send me your advancedsettings.xml in private or post the relevant code here. In reference to the error that you posted previously.
RE: [RELEASE] - Thumbnails Cleaner - User 34959 - 2013-03-18
I use the same advanced setting and did not have any problem, atleast i didn't notice any. And just to confirm, it seems to work fine on mysql table now.
RE: [RELEASE] - Thumbnails Cleaner - DaDuck - 2013-03-18
Quote:If in advancedsettings.xml there is an extraneous char like # (used normally as comment) the script fails because this char isn't part of xml format, for comment in xml uses <!-- -->. I think your problem is this, but if you want, you can send me your advancedsettings.xml in private or post the relevant code here. In reference to the error that you posted previously.
I cleaned up me advanced settings works like a charm now