Kodi Community Forum
Very slow scan library - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Video Support (https://forum.kodi.tv/forumdisplay.php?fid=264)
+--- Thread: Very slow scan library (/showthread.php?tid=367264)

Very slow scan library - Elmout - 2022-03-01

Hi,

I configured Kodi with a shared database stored in my Synology NAS to be able to use it from my TV with Shield TV and my computer and my phone.

My library came from a m3u file witch I split into some strm files. Now I have about 2,500 TV Shows and 8,000 movies to add to my Kodi library.

I tried adding source via SMB protocol and protocol NFS to add all the strm files but the libary scan is very very slow .... In 1 hour only 100 TV Shows are scanned! So to do all my files it will takes days and days! In contrast Plex do this only in severals minutes t!

I tried some data providers (TMDb, TVDB), I try to stop catching tv show rating, and launching the scan from my computer or shield nothing works it's still slow and slow.

What can I do for this works fine ?

Thanks a lot for your help.

RE: Very slow scan library - Elmout - 2022-03-03

I make more tests (disable standby more, reduce number of tvshow, ...) and I thinks that the problem is not the time to scan but that the scan just stop ...

I watch the progress indicator on Kodi and sometimes (after 5/10 tvshows) the progress stop at 100% and never go to the next tvshow ... More severals times Kodi freeze and I have to force stop and restart kodi.

In kodi.log I find some "ERROR <general>: XFILE::CDirectory::GetDirectory - Error getting " but not in the place that kodi stop scanning.

I find another log that attract my attention even it's not a WARNING or an ERROR because it's at the end of the file : INFO <general>: NFS is idle. Closing the remaining connections.

What's the signification of this ? Why it's not an ERROR and why it stop my library scan ?

Thanks for your help.

RE: Very slow scan library - Elmout - 2022-03-03

With the kodi.log file : https://paste.kodi.tv/fukunaquvo.kodi

RE: Very slow scan library - Karellen - 2022-03-03

You did not enable debug mode, so there is nothing of use in the log.

RE: Very slow scan library - Klojum - 2022-03-03

The Python engine/interpreter doesn't seem very happy.

RE: Very slow scan library - Elmout - 2022-03-03

Sorry for the wrong file : 

Part 1 : https://paste.kodi.tv/ibubiferin.kodi
Part 2 : https://paste.kodi.tv/irodabijub.kodi

RE: Very slow scan library - Elmout - 2022-03-05

I look into the log file but don't find any answer on the reason the scan stop ...

Any specialist can help me ?

Thanks a lot.

RE: Very slow scan library - Karellen - 2022-03-05

(2022-03-01, 18:11)Elmout Wrote: Now I have about 2,500 TV Shows and 8,000 movies to add to my Kodi library.

I tried adding source via SMB protocol and protocol NFS to add all the strm files but the libary scan is very very slow .... In 1 hour only 100 TV Shows are scanned! So to do all my files it will takes days and days! In contrast Plex do this only in severals minutes t!
I don't see any problems in your log.
That is an extremely large library. Can you tell me what you are expecting to happen? How fast do you think it should scan? Do you believe that entire library should scan in a couple of hours?
You state 2,500 tv shows, but how many episodes is that - 30,000? 50,000?, 100,000?
You need to expect that yes, scanning a library that large will take a long time. You will need to do it in batches, and for a library that large, ensure you use quite a few sources.
Using stream files isn't helping either. I have a vague memory that Kodi may also be checking the link in the stream file. Why are you using stream files? Do you not have the media files on your NAS?

RE: Very slow scan library - Elmout - 2022-03-08

(2022-03-05, 21:35)Karellen Wrote:
(2022-03-01, 18:11)Elmout Wrote: Now I have about 2,500 TV Shows and 8,000 movies to add to my Kodi library.

I tried adding source via SMB protocol and protocol NFS to add all the strm files but the libary scan is very very slow .... In 1 hour only 100 TV Shows are scanned! So to do all my files it will takes days and days! In contrast Plex do this only in severals minutes t!
I don't see any problems in your log.
That is an extremely large library. Can you tell me what you are expecting to happen? How fast do you think it should scan? Do you believe that entire library should scan in a couple of hours?
You state 2,500 tv shows, but how many episodes is that - 30,000? 50,000?, 100,000?
You need to expect that yes, scanning a library that large will take a long time. You will need to do it in batches, and for a library that large, ensure you use quite a few sources.
Using stream files isn't helping either. I have a vague memory that Kodi may also be checking the link in the stream file. Why are you using stream files? Do you not have the media files on your NAS?

Yes my lbrary is pretty huge but it's "normal". I centralized all movies and tv shows of my family Smile I don't have the media files on my NAS because of the total size.

I don't hope that the scan is very very quick but just don't stop and just finish Smile

RE: Very slow scan library - Klojum - 2022-03-08

(2022-03-08, 11:14)Elmout Wrote: I don't have the media files on my NAS because of the total size.

If the mediafiles are on some cloud service, then checking them for (meta)data will take up some time.
It begs the question, what DO you use the NAS for? Just the MariaDB database?

This is also not helping:
2022-03-03 21:21:55.297 T:49840 DEBUG <general>: DatabaseUtils::GetSortFieldList: unknown field 26
2022-03-03 21:21:55.297 T:49840 DEBUG <general>: DatabaseUtils::GetSortFieldList: unknown field 31
2022-03-03 21:21:55.297 T:49840 DEBUG <general>: DatabaseUtils::GetSortFieldList: unknown field 40
2022-03-03 21:21:55.297 T:49840 DEBUG <general>: DatabaseUtils::GetSortFieldList: unknown field 68
I thought the 10-series of MariaDB was more immune to those. Try doubling any cache settings in the my.ini file. Who knows.

RE: Very slow scan library - Elmout - 2022-03-08

(2022-03-08, 12:25)Klojum Wrote:
(2022-03-08, 11:14)Elmout Wrote: I don't have the media files on my NAS because of the total size.

If the mediafiles are on some cloud service, then checking them for (meta)data will take up some time.
It begs the question, what DO you use the NAS for? Just the MariaDB database?

This is also not helping:
2022-03-03 21:21:55.297 T:49840 DEBUG <general>: DatabaseUtils::GetSortFieldList: unknown field 26
2022-03-03 21:21:55.297 T:49840 DEBUG <general>: DatabaseUtils::GetSortFieldList: unknown field 31
2022-03-03 21:21:55.297 T:49840 DEBUG <general>: DatabaseUtils::GetSortFieldList: unknown field 40
2022-03-03 21:21:55.297 T:49840 DEBUG <general>: DatabaseUtils::GetSortFieldList: unknown field 68
I thought the 10-series of MariaDB was more immune to those. Try doubling any cache settings in the my.ini file. Who knows.

I use my Nas for save the documents : administrative paper for our home, administrative paper for my child's school, wife's business (who is photographers and stock huge stock of picture she made).

I see the sort problem but I have to create the column manually ?

RE: Very slow scan library - Klojum - 2022-03-08

(2022-03-08, 12:56)Elmout Wrote: use my Nas for save the documents : administrative paper for our home, administrative paper for my child's school, wife's business (who is photographers and stock huge stock of picture she made).
Of course...

(2022-03-08, 12:56)Elmout Wrote: I see the sort problem but I have to create the column manually ?
No, I take it all fields in the are there in whatever view it is looking into, but I'm just confused how this happens. It seems like every MariaDB minor version (10.2, .3, .4...etc) has its own hiccups. Either Kodi has a problem building its queries, or something in MariaDB gets messed up.

RE: Very slow scan library - Elmout - 2022-03-09

(2022-03-08, 15:43)Klojum Wrote:
(2022-03-08, 12:56)Elmout Wrote: use my Nas for save the documents : administrative paper for our home, administrative paper for my child's school, wife's business (who is photographers and stock huge stock of picture she made).
Of course...

(2022-03-08, 12:56)Elmout Wrote: I see the sort problem but I have to create the column manually ?
No, I take it all fields in the are there in whatever view it is looking into, but I'm just confused how this happens. It seems like every MariaDB minor version (10.2, .3, .4...etc) has its own hiccups. Either Kodi has a problem building its queries, or something in MariaDB gets messed up.

Why do you doubt about what I'm doing with my NAS ? ...

Ah I try to double cache of mariadb but whitout success.

I try to switch to SMB source and not NFS but don't work either ... I start thinking that Kodi is not the best media center for me ...