[kittmaster]

Thanks. The log eliminated some possibilities and good initiative with the test files, hopefully they help reproduce the issue.

edit: no they don't, the test folder is processed in ~12 seconds regardless of how the files are hosted (local, nas, different protocols). For 22 movies it means half a second per movie, much faster than what you're observing.

One strange thing is how the actors caching doesn't appear in the logs of MainMachine. They did show in the logs of TVC8M2G5-Y700, and also show in my logs and those of neo1973.

NFS protocol has a bit more logging of the timing but not sure it would really help.

My trailer is using SMB and my home network is as mentioned, NFS. But both Matrix and Nexus use NFS also, and have the normal "fast" scan times. I set up Omega the same way and turn off "create video thumbnail" as I don't use those as all the local artwork is present and if it isn't, I don't want a video thumbnail which will show me an issue etc etc....

The Y700 is my laptop (i7, 16GB RAM, SSD, gigabit wifi), the mainmachine is my office (i5, 32GB RAM, SSD, gigabit Ethernet). 

I will try the test folder with the 0byte files to see if I see something different here, maybe something in the MP4 parser?

[kittmaster]

Following the path asked, my y700 got a new 21.1 portable install, downloaded the test files and loaded them on the desktop of Win10 x64, did the install as all previous messages. 

Here is the log:

https://paste.kodi.tv/odikizijur.kodi

It is doing the same thing, 21 seconds per if I'm reading it right.

I'll do the few things to do some testing based on the local files and get back if it yields any fruit.

[kittmaster]

My comment on Github:

My first test: Using test file set you guys have, on desktop unzipped, I turned OFF my WIFI so there was no internet access for it to parse the internet at all and ran the test. It dropped the time per file to around 7 seconds. Here is the log:
https://paste.kodi.tv/ecovusukut.kodi
Next test, Media companion has the option to remove the thumbs URL from NFOs, I'll see if that yields anything.

https://github.com/xbmc/xbmc/issues/25656

[kittmaster]

Here is a thread about the URL, from what was fed back in 2021, it almost seems like Kodi is using the URL vs. local art first. It seems I was asking about this way back when, does this fall into part of this problem?

https://sourceforge.net/p/mediacompanion...165f153f6/

[kittmaster]

Never got the chance to do a bisect on the source code, however, I have a definitive answer to what is happening and should be easy for a fix to be put in back in place.

After some time off, I looked at what was different. Every indication is I have an edge case, and for the most part, that is apparently true.

So with some thinking about it, I downloaded an installed 21.1, ran it the same way, same systems blah blah blah same old result..... mmmkay.

Brainstorm, just install it on my windows machine normally outside of a DEV mode. 

Installed same build, version normally, ran it the same way I always do... and blamo, 1411 movie files went from loading in 9-10 hours to 27 minutes... EXACTLY like Nexus.

So the problem lies exactly with operation of the PORTABLE MODE. I have no idea which of the 8 builds between beta 1 and beta 2 caused this issue but THAT IS WHERE IT LOCATED. 

PORTABLE MODE = OMEGA IS TRASH
NORMAL MODE = OMEGA IS FASTER/BETTER/WORKING

I'm not going to fix this, but I'm pretty there is now enough to look at to fix it once and for all.

This really needs to be addressed, can someone please review this and repair it? 

I'm willing to try a nightly if someone can patch it for testing and hopefully fix makes it to the next build.

Thank you.

[Karellen]

So the problem lies exactly with operation of the PORTABLE MODE

I have four installs running as Portable Mode - v20, v21-stable, v21-nightly, v22-nightly.
Those times I gave you for scanning my library a few months ago were done on Portable installs.

Maybe your hard drive? What type of hard drive is the Portable version installed on and is this different to where the Normal install is located? Mine are running on NVMe SSD

[kittmaster]

So the problem lies exactly with operation of the PORTABLE MODE

I have four installs running as Portable Mode - v20, v21-stable, v21-nightly, v22-nightly.
Those times I gave you for scanning my library a few months ago were done on Portable installs.

Maybe your hard drive? What type of hard drive is the Portable version installed on and is this different to where the Normal install is located? Mine are running on NVMe SSD

I screwed up my regular install because I forgot to add the -p to the shortcut for trying the portable in a different configuration.

However, once I get that patched up, I'll be reupdating with info.

The regular install in on a SSD C:
The portables are on a spinner (very fast spinner) D:

One might say.... that's the problem... it isn't..... the Nexus negation. 

I am trying to put 21.1 portable on my C: SSD, but still working to repair the screwed up startup now, it just crashes... the 21.1 portable and regular install on C.

My other portables are still ok, I tried to uninstall the regular version, but the portable still crashes... GD mess.

I can't understand why you guys can't just give a proper option checkbox during installation to either make it a regular install or portable install and let the install make the correct changes, now I have to clean up a disaster... 

Really getting annoyed.

[izprtxqkft]

I can't understand why you guys can't just give a proper option checkbox during installation to either make it a regular install or portable install and let the install make the correct changes, now I have to clean up a disaster... 

use UniExtract to extract the Kodi installer rather than running it then the installer cannot modify your system

[kittmaster]

I can't understand why you guys can't just give a proper option checkbox during installation to either make it a regular install or portable install and let the install make the correct changes, now I have to clean up a disaster... 

use UniExtract to extract the Kodi installer rather than running it then the installer cannot modify your system

Except for the 2 $ folders, good tip, didn't think of that. worked well. TY.

[kittmaster]

So the problem lies exactly with operation of the PORTABLE MODE

I have four installs running as Portable Mode - v20, v21-stable, v21-nightly, v22-nightly.
Those times I gave you for scanning my library a few months ago were done on Portable installs.

Maybe your hard drive? What type of hard drive is the Portable version installed on and is this different to where the Normal install is located? Mine are running on NVMe SSD

Both of my systems are identical structures, but I have more follow up after this:

On both systems, the C drive is SSD and the D drive is a spinner on both.

I only have access to laptop now, will post other tomorrow when I can but on my i7 laptop, the C drive is: Samsung SSD 990 PRO 2TB and the D drive is: ST2000LM015-2E8174

Both are "new" and are pretty damn fast.... and again... Nexus works correctly as I've been saying on both systems.

===============

So, I did some more testing to verify what may be different. I forced the cache setting to include local files to adaptive, 50x, and various other settings, did not affect anything.

I have a context menu registry addin that can move folders, I did this to the 21.1 Omega and moved it between the C and D drives on this laptop. Without changing anything, on D, slow as usual. I context moved the entire folder to a simple folder C:\appz and then did the rescan and it moves as fast as my recent report where it is about 27-29 minutes to scan all 1412 movies...

So this goes back again to what I reported very very early on when I noticed how slow it is that the D drive on BOTH my systems, the spinner drives are at or near 100% utilization but Nexus does not. As soon as I moved it to C SSD, utilization ranges between 50-90% and the WiFi speeds are north of 10-15Mbps (good) where when on the D drive, it is sub 1Mbps.

It appears the problem is in the file I/O code.

This is something I mentioned long ago. Is there some changelog that can be reviewed between Beta 1 and 2 that talks about any changes in this area? 

And I'll be blunt before someone comes along and says just use the SSD drive..... yeah, no kidding... But let's also realize that not everyone on the planet have SSDs only or they have hybrids like I've done.... so that is NOT the answer unless Kodi's minimum requirements specifically calls out SSDs only and spinner drives not supported..... which we all know, isn't the case... 

So, what can Kodi do to resolve this?

TY.

[Neo1973]

Is there some changelog that can be reviewed between Beta 1 and 2 that talks about any changes in this area? 

https://github.com/xbmc/xbmc/releases/tag/21.0b2-Omega

[scott967]

FWIW, I only run -p for all my installs, and on my dev system from a 15 tb drive with 2 partitions (5/10).  So if I make a mistake just blow away the Kodi %appdata%.  For crashing, my experience when adding a new video source is I can get occasional crashes if I am doing something while video thumbnails are being created in the background (have a few thousand separate MVs, so thumbnailing all those vids takes a while).  Once the video thumbs are complete never a problem.  Not sure on the scan speed from NAS SMB shared source but I don't think it is markedly different between versions (I scan local info only with full nfo and local art).

scott s.
.

[kittmaster]

FWIW, I only run -p for all my installs, and on my dev system from a 15 tb drive with 2 partitions (5/10).  So if I make a mistake just blow away the Kodi %appdata%.  For crashing, my experience when adding a new video source is I can get occasional crashes if I am doing something while video thumbnails are being created in the background (have a few thousand separate MVs, so thumbnailing all those vids takes a while).  Once the video thumbs are complete never a problem.  Not sure on the scan speed from NAS SMB shared source but I don't think it is markedly different between versions (I scan local info only with full nfo and local art).

scott s.
.

All of that is good info, but still, it comes down to someone borked the file I/O and it needs to be fixed. 

To your scan speed statement, I'm using NFS on the NAS over a 2.5Gbps link and as I mentioned, it is scanning at 27-29 minutes for 1411 movies when on the SSD on Omega just as Nexus did, however, Nexus can do it on a standard spinner HDD.... i.e. once again.... someone borked the file I/O between beta 1 and 2 of Omega. This worked correctly under Nexus and prior (tail end of Leia for me), and beta 2 forward it persists to run like a slug on an HDD.

It is silly to expect me to try and troubleshoot this as a Kodi dev, I've identified the error source point location range, failure mode, and hardware metrics.

Personal view: I thought you guys had SVN for a reason to be able to revert back to exact points in time to be able troubleshoot things like this? It's not like being asked to being asked to go back from Omega to Leia and say "what changed"... Someone mentioned to me....there where all these changes.... yeah, so what... SVN tracks all that... that means nothing to me.... guess I should just go on my way then.... Revert, figure it out, fix it, and go back to the current snapshot... what does 1000 changes in between mean or why even both having SVN type control if you don't bother using it to solve problems? I just don't get the mentality; like this is some complicated problem of going back to help solve my "singular" issue..... not like it's something "I" created yet it is hampering everything I'm doing and really demotivating me to even want to continue.

Yes, I'm a little peeved because it seems like it is just because others don't have this issue the all is right with the world... yet let's get to the next letter release... <rant complete>

Other than what I've reported, this will probably be the last post in this thread from me unless an inquiry comes along.

Out.

[CrystalP]

You'll be happy to know that I was able to reproduce the issue with a portable Kodi on a hdd. HDD instead of SSD seems to be the key. The HDD gets maxed out. The location of the movie directory structure you provided doesn't seem to matter.

Unfortunately, a bisection shows that the problem lies somewhere in the huge video versions commit... PR 14972 commit f2ab7a1ed81d8f9632537f1d7b197f2be7121966

[kittmaster]

You'll be happy to know that I was able to reproduce the issue with a portable Kodi on a hdd. HDD instead of SSD seems to be the key. The HDD gets maxed out. The location of the movie directory structure you provided doesn't seem to matter.

Unfortunately, a bisection shows that the problem lies somewhere in the huge video versions commit... PR 14972 commit f2ab7a1ed81d8f9632537f1d7b197f2be7121966

Well thank the heavens..... at least you (and now other devs) have a way to test against this and hopefully find a fix that will work with Omega. 

I will be happy to test any build that addresses this issue to help verify resolution when it becomes available. 

I'm sure it's going to take some time to identify what the cause is, but it is just a relief that you were able to replicate it and I can carry on.

Thanks for looking into it, hopefully it can be easily resolved.

Best,
Chris