+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Supplementary Tools for Kodi (https://forum.kodi.tv/forumdisplay.php?fid=116)
+---- Forum: TinyMediaManager (https://forum.kodi.tv/forumdisplay.php?fid=204)
+---- Thread: Solved TMM scans progressively slower on LARGE TV show database (/showthread.php?tid=176384)
TMM scans progressively slower on LARGE TV show database - IceNine - 2013-10-23
I am giving TMM a try on managing my existing TV shows and found that when I am scanning in my files TMM will progressively use more and more CPU power and get slower and slower. At the beginning of the scan it is running at >30% CPU and I can see the episode names flying by on the status bar, but further into the scan it gets noticeably slower and eventually uses up to 100% of my CPU.
I can stop the scan using the little stop button on the lower right of the window and it stops ok but if I start a new scan right away the CPU usage spikes to 100% and TMM starts dragging again. If I quit TMM and restart it, then start a scan it starts off fast again, but again slows down over time.
I'm guessing it is related to the sheer size of my TV show library (250+ shows / nearly 19,000 episodes) because if I only use the "smaller" of the 2 shares I have TV shows on it will scan fine. If I use the "larger" share or both at the same time is when I run into the issue.
It doesn't even seem like the scan will ever finish, as I started the process for the first time last night and found out my CPU had been maxing out all night long without finishing the initial scan. It was still going, just unbelievably slow at that point.
I also don't see anything wrong on the logs, but the times between pulling MediaInfo for the last file for one episode and moving on to the first file for the next episode get progressively longer there also.
Example:
Code:
From early in the scan:
2013-10-23 08:40:14,533 DEBUG [tmmpool-mediainfo-thread-1] org.tinymediamanager.core.MediaFile:1054 - start MediaInfo for Spaced - S01E06 - Epiphanies.tbn
2013-10-23 08:40:14,963 DEBUG [tmmpool-mediainfo-thread-1] org.tinymediamanager.core.MediaFile:1054 - start MediaInfo for Spaced - S01E07 - Ends.avi
From late in the scan:
2013-10-23 13:53:32,933 DEBUG [tmmpool-mediainfo-thread-1] org.tinymediamanager.core.MediaFile:1054 - start MediaInfo for Doctor Who (2005) - S05E13 - The Big Bang (2).tbn
2013-10-23 13:53:44,631 DEBUG [tmmpool-mediainfo-thread-1] org.tinymediamanager.core.MediaFile:1054 - start MediaInfo for Doctor Who (2005) - S06E01 - The Impossible Astronaut (1).aviAlmost 15 seconds there on the later switch, while the earlier took less that .5 seconds.
EDIT: Worked with mlaggner on the nightly builds, this issue has been resolved!
RE: TMM scans progressively slower on LARGE TV show database - mlaggner - 2013-10-24
thanks for reporting. Unfortunately mediainfo is an external (native) lib, and thus we don't have much space to do something..
Nevertheless we will have a look at this!
RE: TMM scans progressively slower on LARGE TV show database - IceNine - 2013-10-24
I certainly can't be sure that the slowdown is related to MediaInfo at all, that is just the scanning phase where the slowdown becomes noticeable.
If there is anything on my end that you would like me to look for or try on my end please let me know, otherwise I will keep trying to narrow down exactly when/where the issue is occurring.
RE: TMM scans progressively slower on LARGE TV show database - mlaggner - 2013-10-28
we did some enhancements with v2.4.5 - please tell me if it is working better now
RE: TMM scans progressively slower on LARGE TV show database - IceNine - 2013-10-29
It did seem overall faster, but the "increasing CPU usage over time" issue still seems to exist, at least during the initial library import. Full updates after the library has been scraped in seem to be fine now. During the initial full scrape the CPU utilization went from 25% at the beginning of the MediaInfo gathering stage to around 80% towards the end. I did this test on my 'small' share so I will see if it is worse on the 'big' share next. Otherwise the issue does seem to be improved!
RE: TMM scans progressively slower on LARGE TV show database - IceNine - 2013-10-30
When scanning the 'big' share in the issue does seem to be alive and well. It got about 10% into the 'Gathering MediaInfo' stage when CPU usage had reached about 80%. At the beginning of the stage CPU usage was only about 25%. It is a weird issue but it definitely seems to be worse the larger the library is.
RE: TMM scans progressively slower on LARGE TV show database - mlaggner - 2013-10-30
okay,.. but it only affects you at the initial import?
I am afraid this CPU usage could come from mediainfo (which is not fixable by us...); but if it really comes from mediainfo, it will only affect you the first time the files will be scanned.
RE: TMM scans progressively slower on LARGE TV show database - IceNine - 2013-11-01
On further testing using my 'big' share it looks like the issue affects all stages of the scrape of a large library. Scanning and updating the small share appears to be mostly fine, there is still some CPU usage increase but nothing debilitating. It goes from about 25% at the beginning of the update to about 60% at the end. On the larger share CPU usage is at around 70% by the end of the 'scraping' phase (before gathing MediaInfo) and maxes out shortly after that. I started a full TV library update yesterday at 4PM and found it hadn't even finished by 8AM today (and that my CPU had been maxing out all night long.)
RE: TMM scans progressively slower on LARGE TV show database - mlaggner - 2013-11-27
today I've got an idea what is eating up the CPU at your import.
To handle this the right way, we have to rewrite our database access model, which isn't a small part. The good thing is: we already thought about that in the past
I can't give any ETA when this is done.. (since it is a huge part to write, it is much more to test - consistency of our database has highest priority!)
stay tuned
RE: TMM scans progressively slower on LARGE TV show database - mlaggner - 2013-11-27
btw: could you help us finding the source of the problem?
you could send me a complete file listing of your big share, so I can rebuild it with dummy files at home (http://txt.binnyva.com/2007/09/listing-all-files-recursively-in-windows/)
RE: TMM scans progressively slower on LARGE TV show database - roliverosc - 2013-11-29
Hi i have observed that in my case the cpu only climb when download one poster, artwork or fanart from one movie even sometimes looks endless takes too, my collection is around 350 movies. I have a bug in my tinymedimanager but my problem isn't of cpu is with the java process that climbing too high, it´s around 500Mbytes it´s uncanny. I can upload photo if it´s necessary of task manager.
RE: TMM scans progressively slower on LARGE TV show database - IceNine - 2013-11-30
(2013-11-27, 22:27)mlaggner Wrote: btw: could you help us finding the source of the problem?
you could send me a complete file listing of your big share, so I can rebuild it with dummy files at home (http://txt.binnyva.com/2007/09/listing-all-files-recursively-in-windows/)
Sorry for the delayed reply, hadn't been on the boards for a few days. Enjoy the humongous list!
This isn't my entire TV library list which is split across two shares but this is the larger list and seems to be plenty big enough to reproduce the issue.
RE: TMM scans progressively slower on LARGE TV show database - mlaggner - 2013-11-30
thanks for the file listing.
i managed to speed up the import by a factor of about 20 (in my test case the "search phase for 12k episodes took about 1 minute" - Intel Core i5 on a SSD
)Now I have to check the database for consistency and if there are further improvements for the importer
RE: TMM scans progressively slower on LARGE TV show database - IceNine - 2013-12-01
(2013-11-30, 15:45)mlaggner Wrote: thanks for the file listing.
i managed to speed up the import by a factor of about 20 (in my test case the "search phase for 12k episodes took about 1 minute" - Intel Core i5 on a SSD
Now I have to check the database for consistency and if there are further improvements for the importer
Well that certainly sounds promising, keep up the good work! And let me know if there is anything else you need from my end with my huge library.