+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: General Support (https://forum.kodi.tv/forumdisplay.php?fid=111)
+---- Forum: OS independent / Other (https://forum.kodi.tv/forumdisplay.php?fid=228)
+---- Thread: advanced settings.xml question (/showthread.php?tid=372432)
advanced settings.xml question - Jeff721 - 2023-03-09
Good day all!
I have a question regarding my library taking forever to update.
First of all, I have a NAS at home and I have few Android boxes using Kodi. I have created an SQL database on my NAS and all Android boxes have the advancedsettings.xml file in the Kodi folder so all devices connect to one database.
On my NAS, I have a shared folder called Videos and under that folder, I have another folder called Movies, then multiple folders for the type of movies (Action, Drama, Comedy, War, etc) and then one folder per movie. (Videos>Movies>Drama>Movie Name).
I have 3 sources for videos in Kodi and they are set as follows:
- Movies: smb://server_ip/videos/movies
- TV Shows: smb://server_ip/videos/tv_shows
- Family Videos: smb://server_ip/videos/family_videos
After few readings in the Kodi Wiki, I noticed there his an entry available for the advancedsettings.xml file under video library named: usefasthash. This is basically for: !-- defaults to true. Set to false in order to skip hashing based on the folders modification time.
So I made sure this key was set to true but my question is: when Kodi checks for updating the library, how does it work? Does it check at the Source folder (i.e. MOVIES) and then compare its registered hashing to see if something was modified or it does not care and goes to all folders to check if it was modified or not?
In my case, as my file structure is set this way (Videos>Movies>Drama>Movie Name), I want the hashing to be completed/compared at the "Drama" folder, not any other folder. That way, Kodi would only have to check 10 to 15 folders to see if it needs to update the library or not. Right now, even if there has been no updates in my file structure (no new movie added/deleted), when the auto-update of the library runs, Kodi will scan all folders even if the hashing has not changed at all... And it takes forever to complete even if my home network is all 1Gbps based. So I need to know here that hashing comparison occurs... If it is as the SOURCE level, I will create multiple sources (ACTION, DRAMA, COMEDY, etc.) and then will let the hashing comparison to do its magic...
Am I missing something?
Thanks for your help!
Jeff.
RE: advanced settings.xml question - Karellen - 2023-03-10
(2023-03-09, 20:47)Jeff721 Wrote: Does it check at the Source folder (i.e. MOVIES) and then compare its registered hashing to see if something was modified or it does not care and goes to all folders to check if it was modified or not?Sorry, I don't know the definitive answer to your questions, but I am interested to find out. I do believe the fasthash works on the movie folder level, not the source level.
Are you any good at reading code? Maybe start here... https://github.com/xbmc/xbmc/pull/6460
Maybe @mkortstiege can answer if he is around. Maybe @Fuzzard or @lrusak might know?
But there are some workarounds to speeding up the library scanning.
How many sources do you have?
How many of those do you never add new content to? You could mark them as never scan by enabling Exclude selected folder from scans... https://kodi.wiki/view/Adding_video_sources#Set_Content
Do you need to run an Update Library automatically? Are you adding a lot of content to Kodi on a frequent basis?
If not, then you can disable auto scanning on startup, and just manually scan the item you added by finding it in your Source folder in Kodi, and selecting i on your keyboard.
RE: advanced settings.xml question - scott967 - 2023-03-10
I did a quick look-see, and for movies a list of all the movie paths is queried from the library and anything which for various reasons isn't to be scanned is removed. With what's left the path folder fasthash is computed based on folder mtime with fallback to ctime and compared to the hash value stored in the paths table for that path (dbpath). An equal fasthash to the dbhash means do not scan. If the fasthash does not match, the folder is scanned for changes. If there is no fasthash, or advanedsettings has it turned off, then the full folder contents are checked and a "slowhash" computed based on the filenames, filesize and filedate and that is compared to the dbhash.
At least that's how I read it (with no real c++ skills).
scott s.
.
RE: advanced settings.xml question - Jeff721 - 2023-03-11
(2023-03-10, 02:19)Karellen Wrote: But there are some workarounds to speeding up the library scanning.Good day Karellen,
How many sources do you have?
How many of those do you never add new content to? You could mark them as never scan by enabling Exclude selected folder from scans... https://kodi.wiki/view/Adding_video_sources#Set_Content
Do you need to run an Update Library automatically? Are you adding a lot of content to Kodi on a frequent basis?
If not, then you can disable auto scanning on startup, and just manually scan the item you added by finding it in your Source folder in Kodi, and selecting i on your keyboard.
Regarding the amount of sources, it is a tricky question: I now have 3 sources for videos (Movies/TV Shows/Family Videos) but I used to have numerous sources for my movies: one for each type of movie (Action/Adventure/Biography/Comedy/Etc.)... Both way took an eternity to update the Video Library. So I reverted back to the 3 sources...
Those sources all get updated once in a while... Some more often than other but they are still all updated. This is the main reason why I asked at which folder level/source the hashing occurs... Right now, even if I don't change anything, the auto-update (or even the manual update) will go through all sub-folders and look if something has changed. I understand there are ways to "get around" to help speeding up the Library updates but it is kind of defeating the purpose of those options within Kodi. And it does not make sense to put all folders that haven't changed as part of the "exclude list" too...
It would be nice to have the Kodi programmers in this section of the Forum as I would suggest them to change the way the hashing analysis is performed... It would be great if Kodi would be able to keep track of the hashing values of each folders in the library and then compare the values... In my case, if I was to put a new video in the Movies>Drama>Movie Name... Kodi should check the hash at the Movies folder (root), sees the hash has changed, then look at sub folders and check the first one: Action. The hash would remain unchanged then Kodi should skip to the next folder until it reaches Drama. Then the hash would be different and then Kodi should scan all sub-folders under Drama. Proceeding this way would greatly expedite the Library updates.
Hopefully, I'll find a solution that will help resolving my situation!
Thanks for the help!
RE: advanced settings.xml question - Jeff721 - 2023-03-11
(2023-03-10, 04:03)scott967 Wrote: I did a quick look-see, and for movies a list of all the movie paths is queried from the library and anything which for various reasons isn't to be scanned is removed. With what's left the path folder fasthash is computed based on folder mtime with fallback to ctime and compared to the hash value stored in the paths table for that path (dbpath). An equal fasthash to the dbhash means do not scan. If the fasthash does not match, the folder is scanned for changes. If there is no fasthash, or advanedsettings has it turned off, then the full folder contents are checked and a "slowhash" computed based on the filenames, filesize and filedate and that is compared to the dbhash.Good day Scott,
At least that's how I read it (with no real c++ skills).
scott s.
.
I do not know much about programming unfortunately. I think the issue here it that function within Kodi compare the hashing values of ALL entries in the Library (minus the exclusions) rather than doing it sequentially... It should compare the root of the source... If no changes have occurred, then the scan should stop right there and check the next source until it detects a change. If no changes then the Library Update should end. If a sub-folder has changed, this changes should be reflected in the hash values of the root folder. When Kodi would compare the root's hash values to what is in its database/Library, it would see it is different and then it should compare the values of the sub-folders of the root... Then once it detects the folder where the hash values have changes, it should then go to the sub-folders of that folder and repeat the scanning... until all changed values have been captured and the Library is fully updated!
I know it is a very simplistic view on how I think it should work, but I'm no programmer and I am pretty sure it is a lot more complicated than what I said... Maybe something to consider for the next update of Kodi?
Thanks Scott for your help.