Kodi Community Forum
algorithm to calculate imagehash form jpg pictures - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Development (https://forum.kodi.tv/forumdisplay.php?fid=32)
+--- Forum: Kodi Application (https://forum.kodi.tv/forumdisplay.php?fid=93)
+--- Thread: algorithm to calculate imagehash form jpg pictures (/showthread.php?tid=363040)

algorithm to calculate imagehash form jpg pictures - olwaldi - 2021-06-09

I'm using Kodi on a raspberry pi3 to display jpeg pictures. Due to the limited performance of the pi3 the creation of thumbnails is very, very slow. Thus I'd like to create the thumbnails offline on a more powerful hardware.

I have already a prototypic shell script to do that - it can populate the TABLE texture with the columns id, url, and cachedurl (reusing a crc32 algorithem from Milhouse's script TextureCache.py). But I have no idea howto calculate the column imagehash. Only thing I could deduce from inspection of some sample Texture13.db files is that column imagehash follows that pattern: "d" set of digits "s" set of digits like e.g. d1622727716s47601 - thus it seems to be some decimal values (no hex).

Can you point me to the algorithm or source code to calculate imagehash? I additionally assume that the column lasthashcheck contains a datestamp to be created easily.


Thanks a lot, Michael

RE: algorithm to calculate imagehash form jpg pictures - asavah - 2021-06-09

https://github.com/xbmc/xbmc/blob/master/xbmc/TextureCacheJob.cpp#L221-L242 it's basically  d + time + s + size where "time" is file modification or creation time (if mtime is null) in unix timestamp format, "size" is size in bytes

RE: algorithm to calculate imagehash form jpg pictures - olwaldi - 2021-06-09

(2021-06-09, 18:16)asavah Wrote: https://github.com/xbmc/xbmc/blob/master/xbmc/TextureCacheJob.cpp#L221-L242 it's basically  d + time + s + size where "time" is file modification or creation time (if mtime is null) in unix timestamp format, "size" is size in bytes

Thanks a lot, Michael

RE: algorithm to calculate imagehash form jpg pictures - olwaldi - 2021-06-19

In the meantime I've implemented a bash shellscript using sqlite3 and imagemagick to automatically create thumbnails and background images offline. It works as expected (but runs slower than expected even on a more powerful platform ~1s/picture) - but I encounter a mysterious problem: After generation of thumbnails and updating texturedb suddenly ~50% of all my music artwork is missing. To be precise the TABLE texture contains e.g. one row for each mp3 including the hashed path to the (music) thumbnail - as expected, but the thumbnail itself is no longer existing. Thus when browsing half of the mp3 thumbnails are displayed as default folder icon.

Of course my first assumption is to search within my SQL scripting. But the SQL entries all look fine, "somebody" did delete those music thumbnails. All my newly created picture thumbnails are existing.

I've done some statistics - currently there are ~8000 rows within TABLE texture, and ~1000 of them point onto missing thumbnail files. Almost all of these 1000 rows with missing files have an empty entry within column lastdatecheck.

Is there any automatism within kodi to do a kind of automatic garbage collection? Nevertheless I assume a hidden bug within my shell script of course, but till now I had no idea (and I have already fixed a similar bug within my code deleting picture files unexpectedly). But my script does not touch music artwork.

I'm using kodi 18 on a raspberry pi3 and kodi 19 on a linux machine. Thumbnails are shared by NFS, being accessible to kodi by symbolic link
~/.kodi/userdata/Thumbnails -> /bigdisk/Thumbnails
Since sqlite doesn't really support NFS, the Databases are stored as local copies, but also accessed by symbolic links
~/.kodi/userdata/Databases/Texture13.db -> /bigdisk/Texture13.db (local to raspberry pi3).

Of course I'll share my script when I have fixed that annoying problem.

2021-06-19 12:26 addendum: It seems to be that those ~1000 missing thumbnails did vanish already BEFORE I added data from my shell script. Thus I'm quite confident that my script is NOT the root cause of missing thumbnails. Is kodi sensitive if using thumbnails by link into a data area with is even automounted (i.e. mount will happen on first use)?

16:00 addendum2: Probably my loss of thumbnails happened when I did exchange my ancient 320GB drive (many IO errors) by a modern 2TB drive. And I used that chance to go with ext4 formatting. And I guess that due to linux access control I didn't copy all thumbs. I could restore almost all of the missing thumbs still being stored on my old 320GB. Thus my problem is solved. 

Regards, Michael