[m4x1m]

I have not any files in music library, after add some of them, this is database folder content




and this error log after

17:24:46 T:6040 NOTICE: [Thumbnails Cleaner] - Starting Thumbnails Cleaner v1.0.4
17:24:46 T:5964 NOTICE: Thread XBPyThread start, auto delete: false
17:24:46 T:5964 NOTICE: -->Python Interpreter Initialized<--
17:25:09 T:5964 ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
- NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
Error Type: <class 'sqlite3.OperationalError'>
Error Contents: no such table: art
Traceback (most recent call last):
File "C:\Users\HTPC\AppData\Roaming\XBMC\addons\script.thumbnailscleaner\resources\lib\gui.py", line 50, in onInit
Cleaner( self ).getAllInfo()
File "C:\Users\HTPC\AppData\Roaming\XBMC\addons\script.thumbnailscleaner\resources\lib\cleaner.py", line 151, in getAllInfo
match = RawXbmcDb.query( "SELECT url FROM art WHERE media_type='artist' AND type='thumb'", dbMusics )
File "C:\Users\HTPC\AppData\Roaming\XBMC\addons\script.thumbnailscleaner\resources\lib\dbutils.py", line 11, in query
cursor.execute( str )
OperationalError: no such table: art
-->End of Python script error report<--
17:25:09 T:5964 NOTICE: [Thumbnails Cleaner] - Started simulated operation!
17:25:09 T:1168 ERROR: Control 24 in window 13000 has been asked to focus, but it can't
17:25:32 T:5964 ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
- NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
Error Type: <class 'sqlite3.OperationalError'>
Error Contents: no such table: art
Traceback (most recent call last):
File "C:\Users\HTPC\AppData\Roaming\XBMC\addons\script.thumbnailscleaner\resources\lib\gui.py", line 157, in onClick
elif controlId == self.idSimulate: Cleaner( self ).doClean(True)
File "C:\Users\HTPC\AppData\Roaming\XBMC\addons\script.thumbnailscleaner\resources\lib\cleaner.py", line 192, in doClean
self.getAllInfo()
File "C:\Users\HTPC\AppData\Roaming\XBMC\addons\script.thumbnailscleaner\resources\lib\cleaner.py", line 151, in getAllInfo
match = RawXbmcDb.query( "SELECT url FROM art WHERE media_type='artist' AND type='thumb'", dbMusics )
File "C:\Users\HTPC\AppData\Roaming\XBMC\addons\script.thumbnailscleaner\resources\lib\dbutils.py", line 11, in query
cursor.execute( str )
OperationalError: no such table: art
-->End of Python script error report<--
17:29:17 T:1168 ERROR: Control 24 in window 13000 has been asked to focus, but it can't

OK, tomorrow I will release a new version with many fixes (including this)

---

And the video database is set to MyVideos75 in the addon settings, and not one of the older ones?

I would seriously consider removing every database use got and start fresh, your databases have most likely gone so many upgrades that no wonder something is broken.

Those settings are only for mysql users. I found the problem, tomorrow I will post a new version!

[john.doe]

Nice to read it !! if u need previous betatester just say it.

Great work !!

[jmarshall]

m4x1m: Your add-on can't be in the official repository with raw access. Also, it'll break whenever we change the database layout.

Instead, collect together exactly what you need and we'll fix JSON-RPC so that it does everything you need. For the textures database this can just be a list of the queries you use.

Cheers,
Jonathan

[ctawn]

After clicking on Simulate, the buttons are disabled? If not, checks the xbmc log for errors! I don't have any way to test on ATV2. In my system after clicking the Simulate button, all the buttons are disabled to prevent another click, then I can not understand how it is possible to click 3-5 times.

On the ATV2 the buttons are not disabled. It takes a good 30 seconds or more for the notification that the script has started to appear. There's no immediate feedback in other words.

I did go ahead and clean, and it all seems good but for one important thing: all the backgrounds for sections (Weather, Pictures, Movies, etc.) in my skin are not displaying. They are all black. I am using Toiva's Confluence Modified and am using multiple backgrounds for each section. I tried in the skin settings to reset the image folders, but it doesn't work. It's as if it is expecting the cached images and can't find them. I'll post more in that thread about this. But for this cleaner, it might need to take into account skin images, no?

[User 34959]

m4x1m i don't think there's anything i can do on the skin side to fix this.

Most of the skins support using custom backgrounds in home and library as backdrops. My implementation should be identical of other skins.

ctawn, you could try backing up your texturesX.db, deleting it and and letting XBMC recreate the database, it will take a while though if you have a huge library as it'll need to regenerate every thumbnail, if i remember correctly.

[Milhouse]

ctawn, you could try backing up your texturesX.db, deleting it and and letting XBMC recreate the database, it will take a while though if you have a huge library as it'll need to regenerate every thumbnail, if i remember correctly.

You can use this script to pre-load your cache with all of your artwork. You can also use that script to query your texture cache (using the "s" option) to determine what these multiple images are called, and perhaps identify why they have gone AWOL. I use Confluence Modified myself, and the default Confluence Modified backgrounds (single images - tvshows.jpg, movies.jpg, settings.jpg etc.) don't appear to be cached (./texturecache,py s skin.confluence.modified" returns nothing relevant).

The script will also "prune" the texture cache, equivalent to this thread but using a slightly different technique and is of course command line based, so you could schedule it to run regularly etc.

[Milhouse]

Instead, collect together exactly what you need and we'll fix JSON-RPC so that it does everything you need. For the textures database this can just be a list of the queries you use.

Cheers,
Jonathan

If you're looking for JSON-RPC calls for Texture*.db:

1) Method to search the texture table (joined with sizes table) and return all rows and columns, or a filtered subset
2) Method to delete rows from database tables (using texture.id), with optional deletion of associated filesystem (cachedurl) object
3) Method to get list of all files present in the Thumbnails folder
4) Method to delete files that are in the Thumbnails folder

[ctawn]

m4x1m i don't think there's anything i can do on the skin side to fix this.

Most of the skins support using custom backgrounds in home and library as backdrops. My implementation should be identical of other skins.

ctawn, you could try backing up your texturesX.db, deleting it and and letting XBMC recreate the database, it will take a while though if you have a huge library as it'll need to regenerate every thumbnail, if i remember correctly.

If not the skin, shouldn't XBMC automatically regenerate thumbnails from anything that's missing in the cache??

Regarding the aforementioned python script, can that be run on the ATV2? (No idea if python is installed on that.) If so, how could it be scheduled on an ATV2? (I don't know if cron is installed or how to configure it etc.)

Regarding deleting texturesX.db, which one do I delete? I have textures6.db and textures13.db. Or do I delete both? I guess I'll just try deleting one at random and see what happens first…

[m4x1m]

m4x1m: Your add-on can't be in the official repository with raw access. Also, it'll break whenever we change the database layout.

Instead, collect together exactly what you need and we'll fix JSON-RPC so that it does everything you need. For the textures database this can just be a list of the queries you use.

Cheers,
Jonathan

A method to query all the season arts and the episode arts without use the tvshowid. (to avoid cycles that would delay the execution)
A simply method to query all the actor thumbs (now I use SELECT url FROM art WHERE media_type='actor' AND type='thumb')

For texture db I would to query and delete all fields with lasthashcheck empty (begins with http:// url for example thumbnails created by youtube plugin) and I would query and delete all fields that are older than X days but in the future I would query the url field with a LIKE %usertext% style for eventually delete the results.

On the ATV2 the buttons are not disabled. It takes a good 30 seconds or more for the notification that the script has started to appear. There's no immediate feedback in other words.

I'll try to add a better notification.

m4x1m i don't think there's anything i can do on the skin side to fix this.

Most of the skins support using custom backgrounds in home and library as backdrops. My implementation should be identical of other skins.

ctawn, you could try backing up your texturesX.db, deleting it and and letting XBMC recreate the database, it will take a while though if you have a huge library as it'll need to regenerate every thumbnail, if i remember correctly.

If not the skin, shouldn't XBMC automatically regenerate thumbnails from anything that's missing in the cache??

Regarding the aforementioned python script, can that be run on the ATV2? (No idea if python is installed on that.) If so, how could it be scheduled on an ATV2? (I don't know if cron is installed or how to configure it etc.)

Regarding deleting texturesX.db, which one do I delete? I have textures6.db and textures13.db. Or do I delete both? I guess I'll just try deleting one at random and see what happens first…

Current in use is texture13.db. From my experience I know that delete the texture db does not affect the poster, fanart, thumb etc, but involves the logo, clearart, extrafanart etc that are regenerated when is needed, but I could be wrong.

[User 34959]

Now that i'm testing different cleaning settings i have to say that i don't understand at all how this addon works.

If i use the default settings and run the simulation it says it could save 11MB of 730MB. All the files it has copied are clearlogos of movies and tvshows that are still in my library. I have about 188 movies and 50 tvshows and there are 104 clearlogos that it wants to delete. You are aware that new versions or Artwork Downloader stores the urls to fanart.tv in the art table, and that artwork is then cached locally? Should be easy to check if the extra artwork is still needed by something in the video library or not i guess.

If i set "remove older than..." to 0 which should be disabled it says i can save 356MB. This time wants to delete addon thumbs that are still installed, fanart and clearlogos for videos that are still in the library.

If i untick the lasthashcheck option the result is very close to the one above, addons icons, fanart, clearlogos.

Also it's very easy to crash the add-on by removing the configured output directory.. Ok this is clearly a user error but maybe it should check that the selected directory actually exists? Also, i changed the output directory to something else and ran the cleaning process, but it did not use the new directory.

Code:

<settings>
    <setting id="dbMusics" value="32" />
    <setting id="dbTextures" value="13" />
    <setting id="dbVideos" value="75" />
    <setting id="destinationFolder" value="/Users/anssi/Pictures/clear/" />
    <setting id="emptyTable" value="" />
    <setting id="oldDays" value="30" />
    <setting id="processTDB" value="true" />
    <setting id="removeEmpty" value="true" />
</settings>

Code:

ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
                                             - NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
                                            Error Type: <type 'exceptions.IOError'>
                                            Error Contents: (2, 'No such file or directory', '/Users/anssi/Pictures/test/')
                                            Traceback (most recent call last):
                                              File "/Users/anssi/Library/Application Support/XBMC/addons/script.thumbnailscleaner/resources/lib/gui.py", line 156, in onClick
                                                if controlId == self.idClean: Cleaner( self ).doClean(False)
                                              File "/Users/anssi/Library/Application Support/XBMC/addons/script.thumbnailscleaner/resources/lib/cleaner.py", line 310, in doClean
                                                self.finalizeClean( doSimulate )
                                              File "/Users/anssi/Library/Application Support/XBMC/addons/script.thumbnailscleaner/resources/lib/cleaner.py", line 348, in finalizeClean
                                                shutil.move(c[1], destinationFolder)
                                              File "/Applications/XBMC.app/Contents/Frameworks/lib/python2.6/shutil.py", line 264, in move
                                                copy2(src, real_dst)
                                              File "/Applications/XBMC.app/Contents/Frameworks/lib/python2.6/shutil.py", line 99, in copy2
                                                copyfile(src, dst)
                                              File "/Applications/XBMC.app/Contents/Frameworks/lib/python2.6/shutil.py", line 53, in copyfile
                                                fdst = open(dst, 'wb')
                                            IOError: (2, 'No such file or directory', '/Users/anssi/Pictures/test/')
                                            -->End of Python script error report<--
16:24:48 T:140735215563136   ERROR: Control 24 in window 13000 has been asked to focus, but it can't

Last but not least, after running the cleaner i have to browse through movie and tvshow library to regenerate all the posters and fan arts, why? It was not supposed to touch those.

If you don't use Artwork Downloader yourself and want to know what is stored in the table, let me know, i can help you.

[ctawn]

How long could XBMC take to regenerate movie fanart and thumbs after deleting the textures13.db? Xbmc has been running for like 4 hours and my movie library of about 400 movies is only partially done it seems.

Tv shows mostly done (20 or so shows) and no indexed music.

edit: Do I need to manually browse every movie for example? Because just selecting it from the library list doesn't seem to work, at least not right away (or close to right away).

[m4x1m]

If i use the default settings and run the simulation it says it could save 11MB of 730MB. All the files it has copied are clearlogos of movies and tvshows that are still in my library. I have about 188 movies and 50 tvshows and there are 104 clearlogos that it wants to delete. You are aware that new versions or Artwork Downloader stores the urls to fanart.tv in the art table, and that artwork is then cached locally? Should be easy to check if the extra artwork is still needed by something in the video library or not i guess.

I didn't know! Thanks

If i set "remove older than..." to 0 which should be disabled it says i can save 356MB. This time wants to delete addon thumbs that are still installed, fanart and clearlogos for videos that are still in the library.

Yes, I know. Fixed!

Also it's very easy to crash the add-on by removing the configured output directory.. Ok this is clearly a user error but maybe it should check that the selected directory actually exists? Also, i changed the output directory to something else and ran the cleaning process, but it did not use the new directory.

Fixed!

Last but not least, after running the cleaner i have to browse through movie and tvshow library to regenerate all the posters and fan arts, why? It was not supposed to touch those.

If you don't use Artwork Downloader yourself and want to know what is stored in the table, let me know, i can help you.

I use Artwork Downloader! I noticed ...

Movie -> clearlogo
TvShow -> clearlogo and clearart

I don't know if there are others!

UPDATE
media_type -> movie, set, tvshow, season, episode, musicvideo or actor
type -> poster, fanart, thumb, banner, landscape, clearlogo, clearart, characterart or discart

OK, I have to change the approach! Please be patient.

[ctawn]

My understanding of any of this thumbnail / image caching is very limited and it looks like y'all have some issues to work out that is beyond me.

All I can say at this point is that I regret that XBMC isn't able to internally handle cached images in a better way. It seemed to me quite bad – given the ATV2's limited storage space – that there were nearly 2 GB of cached images for only a few hundred movies and TV shows. So I ran the cleaner and that saves the space but now I'm spending a good part of a day cleaning up the mess (getting back lost images, etc.).

I hope JM or another dev can look into this need again for the future.

[jmarshall]

ctawn: just make sure you don't have any old images left over from pre-Frodo - that's usually where they all are. Anything not in the root of Thumbnails/<a-f0-9> can be deleted (excluding Thumbnails/Video/Bookmarks if you want to keep them).

Also, as you've deleted Textures13.db, the only way to get art back is to browse one by one (or run MilhouseVH's script to repopulate it).

[ctawn]

jmarshall: thanks.

FYI, I have a backup of textures13.db and could revert if necessary. I have been browsing one by one but it seems to take a heck of a long time to get the artwork back. In fact, I assumed it was working in the background because when I come back to it, some is repopulated and others are not. Maybe it had just eventually caught up to ones I browsed.

I have no idea how you could tell what is pre-Frodo. Unless I guess you're saying /Thumbnails/generated ../Music, ../Pictures, ../Profiles, ../Programs, and ../Video (except ../Video/Bookmarks/) is all pre-Frodo??

What about a bunch of old .db files?

It would be real swell if XBMC had these clean up functions built in. I guess I will attempt to run millhouseVH's script. (edit: I did but got an error immediately.)