+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Supplementary Tools for Kodi (https://forum.kodi.tv/forumdisplay.php?fid=116)
+---- Forum: TinyMediaManager (https://forum.kodi.tv/forumdisplay.php?fid=204)
+---- Thread: Solved Updating Movies sources takes *forever* (/showthread.php?tid=175487)
RE: Updating Movies sources takes *forever* - mlaggner - 2013-10-13
it should...
at least here in linux it is running in a 64bit JVM
RE: Updating Movies sources takes *forever* - Axodious - 2013-10-13
(2013-10-13, 09:25)mlaggner Wrote: extract the dll(s) from the download you need and extract it to the corresponding folder in the tmm install install dir under native
e.g.
tmm\native\windows-x64\MediaInfo.dll
I believe the dll should be in tmm\native\windows-amd64\ for 64bit, at least that's where mine is.
RE: Updating Movies sources takes *forever* - IceNine - 2013-10-14
Things are certainly improving but still slower than I think it should be over here. Right now a full update will take just about 8 minutes. I tried 3 separate times and get very consistent times. The breakdown of time taken goes:
'Big' share scan - 2 minutes
'Big' share MediaInfo / cleanup - 4 minutes
'Small' share scan - 40 seconds
'Small' share MediaInfo / cleanup - 1 minute 20 seconds
I tried both the existing and newer versions of the MediaInfo DLL and get the same times. The very first scan I did after the update did take longer but after that it is very consistent. I also see in the logs that the .srt files that seem to take so long only scanned the first time after the update and now are not scanning anymore, so that seems to have saved a lot of time. It still seems strange to me that the MediaInfo / cleanup stage still takes so long, especially since in the logs it doesn't even seem to be doing that much anymore. No files are scanned for MediaInfo and this is the only log entry I get for that stage:
2013-10-14 09:15:05,782 INFO [SwingWorker-pool-2-thread-5] o.t.c.m.tasks.MovieUpdateDatasourceTask:147 - removing orphaned movies/files...
That entry takes up the entire 4m + 1m20s for both shares. I can't say I know exactly what it is doing during that stage but it seems strange that it would take twice as long as the scanning phase.
Comparing my update times to mlaggners above, it seems like my times would be mathematically right where they should be if the MediaINfo/cleanup part didn't take so long. He has 500 movies to scan and is getting about 30 seconds on an update with no media added, with 2000 movies it should logically take around 2 minutes which is pretty much what my times are without counting MediaInfo/cleanup.
Still not sure what is taking so long but I am going to try to see if I can rule out my PC or OS as the issue. I'll run some scans on the new version on my Mac when I get a chance, and I also plan to set up some clean Linux and Win7 virtual machines to see if TMM runs faster off the same sources there. I'll report my findings when I can, thanks again for all the hard work!
RE: Updating Movies sources takes *forever* - myron - 2013-10-14
Well, i see a great improve, based on your findings.
Overall, we're 7 times faster now ^^
14 min -> 2 min
28 min -> 4 min
2 min -> 40 sec
12 min -> 1,5 min
Cleanup is very I/O consuming - checking if every file/folder physically exists.
2000 movies with... ~30 files each (? - depends on your settings, amount of fanarts , etc) we have easily 60.000 files to check.
5 mins for that is really nice i guess
Dunno if we can get more improvements here (need to add some logging)
RE: Updating Movies sources takes *forever* - IceNine - 2013-10-14
I definitely agree there has been a huge improvement. I'm just curious on why my times are so much longer, comparatively, to mlaggner. For a 500 movie source, which is his testbed and my 'Small' share, he is getting 30 seconds update time when there have been no additions and I am getting almost 2 minutes. If I could speed it up on my end to be 4x faster than it is now that would be fantastic.
I just got my Linux VM up and running and the sources are scanning in now, I'll have to see if there is any major speed difference between Linux and Win7.
Also, I don't think I should have any huge amount of extra files over anyone else. Each movie folder has the video file, 1 poster, 1 fanart and the nfo file, with a very small amount of movies also having an srt subtitle file, so there are really only 4 files typically in each movie folder. I'm not sure what 30 files more I would even put in there.
RE: Updating Movies sources takes *forever* - mlaggner - 2013-10-14
could you have a look at the logs again?
one thing there is interesting for me: the time between 2 logs containing the "- PAH - normale movie directory"
here is an example from my PC:
2013-10-13 20:08:25,773 DEBUG [tmmpool-update-thread-2] o.t.c.m.tasks.MovieUpdateDatasourceTask:399 - PAH - normal movie directory:...
2013-10-13 20:08:25,811 DEBUG [tmmpool-update-thread-2] o.t.c.m.tasks.MovieUpdateDatasourceTask:399 - PAH - normal movie directory:...
as you see here, it took 40ms! (accessing the movie directories an my NAS via GVFS/SMB on a 1Gbit LAN). If you have a larger difference of the timestamps, that will indicate that the NAS is either responding slow or something else throttles the connection to the server.
with a "slower" network connection to the server, the cleanup tasks will indeed take much longer (because every file in you database is cross checked with the one on the server). MI will only be scraped once with the latest release
hth
Manuel
RE: Updating Movies sources takes *forever* - IceNine - 2013-10-14
Well I just finished my round of testing on the Linux VM and I don't see any improvement there, so I don't think the slowness I am getting is Windows-specific. It was actually quite a bit slower on the VM, but there is going to be an expected performance hit anyway. The ratio of time spent on scanning phase to MediaInfo and cleaning phase was pretty much exactly the same though, so I would say the results are as expected.
It is certainly possible that the current times I am getting are limited by my NAS, I will look into anything I can do to optimize it. Either way it seems like it is probably as fast as it will go right now, but it MUCH better than when I opened the thread, so I want to thank all the devs again for all your work. If you ever need a 2000+ movie / 18,000+ TV episode testbed just let me know.
RE: Updating Movies sources takes *forever* - mlaggner - 2013-10-14
thanks for your patiente (and tenacity
) - otherwise we wouldn't find these small issues..Just a few minutes ago, I found out that one small change from yesterday had a higher (positive) impact than I thought. And now I am testing an other possible enhancement to the "cleanup task". But without pointing us into the right direction, we wouldn't find these issues (since it's fast here at home
)
RE: Updating Movies sources takes *forever* - IceNine - 2013-10-14
I certainly have enjoyed the process. I may not be much help with code but I would like to think I am a fairly good bug-finder/tester. Keep up the optimizations and I will keep bringing up new ideas.
RE: Updating Movies sources takes *forever* - IceNine - 2013-10-15
I did have another thought while looking into any further optimizations I can do on my end. This idea is completely a "maybe someday" kind of idea and I have no expectations of it being implemented soon if at all, but have you thought about multi-threading TMM? I'm sure that is a much bigger project than even I think it would be, but while running tests it looks like only 1 core of my CPU is being used when updating the TMM library. I have a quad-core which I'm sure not everyone will have, but I would imagine most PCs these days have at least a dual core, so multi-threading seems like a reasonable way to possibly double the performance of some of TMMs operations. But again, this is more of a 'thinking out loud' kind of thing, feel free to tell me no way.
RE: Updating Movies sources takes *forever* - mlaggner - 2013-10-15
we are already multi threading everywhere where it makes sense
RE: Updating Movies sources takes *forever* - Axodious - 2013-10-15
Regarding the multi threading I think if you're using the 32bit version of Java that it's only running a DB update on 1 thread, I don't have time at the moment but I will e-mail logs running between 32bit and 64bit. When I switched to 64bit I started seeing 3 threads for the DB update whereas only 1 on the 32bit version.
RE: Updating Movies sources takes *forever* - IceNine - 2013-10-15
I am running on Win7 64-bit, I thought I was running 64-bit Java as well but I will have to double check.
RE: Updating Movies sources takes *forever* - IceNine - 2013-10-15
Yup, I see the 3 threads running in the logs as well. I guess my CPU is not the limiting factor. Any chance of being (optionally, of course) able to increase the number of threads in the future? I'm not sure if it would speed things up for me or not but I seem to have CPU to spare. Just trying every angle to get TMM running as fast as possible.
RE: Updating Movies sources takes *forever* - mlaggner - 2013-10-15
we changed back to 3 thread with the last release
it has nothing to do with the architecturemost of the time (especially with file access) the limiting factor is the IO (HDD, Network, ..)
iirc there is only 1 operation in tmm which benefits from multi processor (image caching) - all others are limited by IO. And like I said before - we already handle different tasks with more than one thread - but moving to more threads does not solve everything