[Elusity]

Hi,

Maybe its already there, but I could not find it. Lately I've been adding to my library and need to scan more frequently. The scanner just takes forever, so I update manually by sorting on "date added with descending" and then select "add to library" on the top X items in files. Is there a way to configure the library updater to do this by default ?

If not, I would hereby like to request a quick/partial scan/update function that does this.

Thank you

[Hitcher]

I don't pretend to know how it works but surely it has to scan everything to know if something is new or not?

[jjd-uk]

Update on the side menu should only do new stuff as it will only scan folders where the hash has changed. If the hash is unchanged the folder will be skipped.

[Elusity]

I don't pretend to know how it works but surely it has to scan everything to know if something is new or not?

No, it could just log the timestamp of the last scan. Then process all folders newer than said timestamp.

Update on the side menu should only do new stuff as it will only scan folders where the hash has changed. If the hash is unchanged the folder will be skipped.

Thus it checks the hash for every folder every time. Maybe for the initial scan or bi-yearly update. But for a regular update, that just does not make sense.
Let say you have 100 folders and 1 of them is new. Why would you get the hash for those 99 others in order to skip them?
Just log the timestamp of the last scan. Then process all folders newer than said timestamp.

Hence my request

[jjd-uk]

What happens if a user replaces file, for example replaces the HD version with a UHD version? what happens if they add new local artwork? The only way of knowing that is checking the hash which is a very minimal overhead.

[Hitcher]

No, it could just log the timestamp of the last scan. Then process all folders newer than said timestamp.

I didn't mean scan as in by the scraper but as in Kodi needs to check that each folder hasn't changed since the last time.

[Elusity]

What happens if a user replaces file, for example replaces the HD version with a UHD version? what happens if they add new local artwork? The only way of knowing that is checking the hash which is a very minimal overhead.

In all your examples the timestamp of said folder changes. (no matter the change inside). That timestamp can be used as a stop condition.

1. Sort the queue on date with reverse sort (newest first)
2. Process the item by checking the hash
3. When a timestamp of a folder is older than last scan/scrape: stop processing.
4. Write the scrape completion timestamp somewhere in the Kodi database.

This is the reason why it can be done faster manually and why the scan/scrape time can be improved significantly.
 

No, it could just log the timestamp of the last scan. Then process all folders newer than said timestamp.

I didn't mean scan as in by the scraper but as in Kodi needs to check that each folder hasn't changed since the last time.

No it does not have to. If you reverse sort on date you can stop processing on the first folder that is older.

[ashlar]

What happens if a user replaces file, for example replaces the HD version with a UHD version? what happens if they add new local artwork? The only way of knowing that is checking the hash which is a very minimal overhead.

I'm not sure about the whole timestamps handling (especially as it would apply to all the different OSes/sources that could be in use) but, trying to follow OP's logic, even admitting that in the cases you describe the timestamps would not change... then you'd select the "full scan" option (which would work exactly as it does now) and not the "fast scan" one. Does it make sense?

[jjd-uk]

Hash checking is fast scanning, if anything checking and comparing timestamps will be slower.

[jbinkley60]

The ability to automatically detect  new / changed media could be a very valuable feature for Kodi.  Here's an example of how Mezzmo implements this feature where I can set the time interval to check for new / changed  content by folder.  Having used this type of a feature on a very large library I have ended up setting some folders similar, to how they suggest, with short timer intervals where they are receiving PVR recordings and I want to add them to my library quickly (~2-3 minutes) and there are other folders which rarely, if ever change and I have this feature disabled. 

Current I sync Mezzmo to Kodi to update the Kodi library.  My suggestion would be is that if this is implemented in Kodi natively to provide the user with the flexibility to set refresh timers by folder vs. a fixed interval global  timer for the entire library.  It might be as simple as adding two columns in the path table one to hold the retry timer interval and the second to hold the value of the last time the folder was checked.. 

My 2 cents.


Jeff

[HomerJau]

What happens if a user replaces file, for example replaces the HD version with a UHD version? what happens if they add new local artwork? The only way of knowing that is checking the hash which is a very minimal overhead.

In my experience for Kodi’s video library it never updates updated images automatically in that situation. I need to run some external code to update artwork since Kodi uses its texture cache (local artwork) not images from the Source folders. I’m pretty sure same for music library if you change an album cover.

For speedier library scanning: For movies & TV Series I use local nfo files created by TinyMediaManager. When I scan my video folders (I have 2 x NAS) it’s extremely fast as no internet scrapping is involved, only data from the NFO files. I also use Intel NUCs with NVME SSDs and 2.5GBE Ethernet and NAS volumes are RAID 5. All my movies, TV Series and music videos are scanned and new content added in less than 1 minute (>120TB video storage).

[Elusity]

What happens if a user replaces file, for example replaces the HD version with a UHD version? what happens if they add new local artwork? The only way of knowing that is checking the hash which is a very minimal overhead.

I'm not sure about the whole timestamps handling (especially as it would apply to all the different OSes/sources that could be in use) but, trying to follow OP's logic, even admitting that in the cases you describe the timestamps would not change... then you'd select the "full scan" option (which would work exactly as it does now) and not the "fast scan" one. Does it make sense?

Tagging along that the "descending" and the "date added" functionalities are already in Kodi

Hash checking is fast scanning, if anything checking and comparing timestamps will be slower.

Then how come I can update faster manually?
Secondly you speak as if date checking is a replacement for hash scanning, its not. It still needs hash scanning.
And lastly could you answer my previous question please?

Let say you have 100 folders and 1 of them is new. Why would you get the hash for those 99 others in order to skip them?

You do not need to check "all items" to see whats new is the point I am making.
In this example it only has to sort once and then process 2 items instead of a 100.

[Hitcher]

An alternative is to keep new movies in a separate folder and in the scraper settings for your main movie folder enable 'Exclude path from library updates', this way only the new movie folder will get get scanned by a library update.

[Elusity]

An alternative is to keep new movies in a separate folder and in the scraper settings for your main movie folder enable 'Exclude path from library updates', this way only the new movie folder will get get scanned by a library update.

Thanks for thinking along, I'm doing it manually now. Its not that bad, just very inconvenient. I think your workaround works in the short run, but will get slower as time progresses. Plus I have to move the content somewhere in the future, then scrape again or go in with SQLlite to change paths.

Just waiting on jjd-uk to answer my question, let see where it leads.

[PatK]

Entering file mode or sources, right mouse click on the source in focus will produce a context menu option with 'Scan for new content', so it's not necessary to use exclude path for other sources, if you don't allow global scan updates. Hitting the keyboard 'i'nformation feature in file mode on an individual addition will activate the scraper for new files.