The ethernal caching issue - source too slow and other buffering and stuttering...

The ethernal caching issue - source too slow and other buffering and stuttering... - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: General Support (https://forum.kodi.tv/forumdisplay.php?fid=111)
+---- Forum: OS independent / Other (https://forum.kodi.tv/forumdisplay.php?fid=228)
+---- Thread: The ethernal caching issue - source too slow and other buffering and stuttering... (/showthread.php?tid=364901)

The ethernal caching issue - source too slow and other buffering and stuttering... - pa-pa-americano - 2021-10-12

Hello,

This post concerns mainly Kodi Leia, but not only, it might be useful for one lower or one higher release at least

I come here as the last resort, I think I've read most if not all the posts about kodi caching video files so let me try to explain
I post it in OS independent because I don't believe it has something to do with my OS (which is LE9.2)

I am a kodi user for several years now, I had kodi on windows and linux, with intel and arm arch processors

Mostly, I used kodi on RPI2, then RPI3, now RPI4 because I watch videos that gets heavier and heavier
I could watch a .avi video on RPI2, now I run RPI4 with .mkv movies over 30GB regularly

Now the introdution is done I'll setup the initial condition, the goal and the issues encountered

I have a hardware capable of running Kodi alone ( with a minimalistic OS underneath it of course )
My network is a gigabit LAN over ethernet
The video sources are pointing to an SMB share such as:

| KODI | <----- ethernet cable 1000Mbps -----> | Files sharing server |

The goal is to be able to play a single ~40-50 Gigabyte .mkv file containing a 2 hours movie on Kodi without buffering and stuttering
I'd start the movie and hit the "Pause" button to let Kodi cache it to RAM or most probably a disk (external HDD, SSD, SD, whatever)
My hardware is capable of decoding the codecs of such files and the network is obviously not a problem, I have dedicated ethernet cable for that

The constraints:
The file server is capable of sharing to any network equipment a speed up to 5 Megabytes per second

Now if my calculations are right, if I get the max upload from the file sharing server of 5 Megabytes per second, I could watch a movie of (5 * 60 * 60 * 2 / 1024) 35 Gigabytes for 2 hours without requiring advanced caching, which is far from the 50 Gigabyte movie required
In my case, I want to cache 7Gb of that movie in the RAM or even more on the disk (by pausing the playback at the beginning and come back and play it after caching)

Now let's go through the wikis and posts on forums:

I had no success configuring my Kodi with this wiki:
https://kodi.wiki/view/HOW-TO:Modify_the_video_cache

Settings used:
<buffermode>1</buffermode> : Buffer all filesystems, both internet and local
I tried also to write only 4 but no changes

<memorysize>894784853</memorysize> : The number is the buffer size in bytes (20971520 is the default, which is 20MB, which requires 60MB of free RAM)
So I run it on a RPI4 with 4Gigabytes of ram, then I get the free ram available from the debug statistics obtained from the player with the hotkey CTRL+SHIFT+O which would be around 2 and a half Gigabyte, divide by 3 and write the bytes it in the config file:
>>> 2.5/3*1024*1024*1024
894784853.3333334
894784853 that would be the ideal RAM to put on cache but! simple experiment, run the following command in the terminal just before starting the playback (on linux):
$> watch -n 1 'free -m'
this should show us how much RAM is left in Megabytes.
After starting the playback and pausing for caching, the RAM almost doesn't fill, +- 20Megabytes, that's all...
So I tell myself, sh*t, I misplaced a number or misunderstood something in the documentation, so I tried 89478485300, 8947848530, 89478485, 8947848, 894784, 89478, you get the idea... the setting doesn't do anything, so maybe if my RAM is not compatible something something kodi or linux version dependent bla bla, I disable it by writing 0 in the config file and monitor the filesystem with the following command:
$> watch -n 1 'df -h'
I must see the FS available space to reduce because I disabled RAM cache so it should exclusively write caches on disk, and I wait, I wait, I wait..., after 15 minutes of waiting and seeing no changes I try to play with one last setting

<readfactor>10</readfactor>: Increase the fill-rate of the cache
I did extensive tests with readfactor of 1, 2, 4, 10, 20, 50, 100, including going over all the values mentionned above over and over again

After hours of tests, I conclude that the advancedsettings.xml file has no effect on caching, but hey, maybe I just did everything wrong? So we continue to the next step

OpenWizard addon:
https://www.kodi-guide.com/how-to-fix-kodi-buffering/

Now that I tried to fix the things manually, I decided to use this addon, long story short, it has no effect on caching, the only good thing this addon did for me, is cache folder cleanup from time to time
This addon sets the advancedsettings.xml file to most optimal values if you select the right options (I tried them all)
Conclusion: sounds good, doesn't work

Some other links worth mentionning:
https://forum.kodi.tv/showthread.php?tid=221134
The guy had slow NFS on Kodi and got a workaround by mounting the NFS share with a script instead of relying on the Kodi media source
I tried to mount my SMB share with a command line and copy a single file from remote to local and measure the transfer speed
Conclusion: manually mounted SMB share or set up through media sources in Kodi GUI have same speed more or less

https://www.reddit.com/r/kodi/comments/q4o6s8/source_too_slow_messages_when_playing_uhd_videos/
VLC caches the SMB shares better than kodi, the person seem to have a similar issue as me
I tried to fiddle with VLC Network caching settings, it works in milliseconds and not bytes, so I can write up to one minute of network caching which would make at my download rate, an advance of only 240 Megabytes, not that much but at least it works.
Conclusion: VLC Network caching settings functions as expected even if it's a bit limited, I could maybe try to recompile it and remove the max one minute setting network cache limit to get what I need, or find the config file responsible for it and write it here (might not work as I cope tho...)

https://forum.kodi.tv/showthread.php?tid=335626
This person tries to fiddle with the advancedsettings.xml like me
Conclusion: not much success for him...

Other 2 links without solution for me:
https://forum.kodi.tv/showthread.php?tid=324312&highlight=memorysize
https://forum.kodi.tv/showthread.php?tid=314480

Conclusion:

I am not the only one with that issue as I can see, normal video players allow caching on disk or RAM without issue, and through native UI, not some config files
Now if it's too much to ask to have a decent caching system, let me know, I am ready to accept that my expectations are too high here

If I don't find the solution on this forum, I might consider abandonning running Kodi on Raspberry Pi, or running Kodi on LibreElec, or running Kodi, forever, I've spent too much time on that bullsh*t

Regards

RE: The ethernal caching issue - source too slow and other buffering and stuttering... - Karellen - 2021-10-12

(2021-10-12, 03:43)pa-pa-americano Wrote: OpenWizard addon:
https://www.kodi-guide.com/how-to-fix-kodi-buffering/

Now that I tried to fix the things manually, I decided to use this addon, long story short, it has no effect on caching, the only good thing this addon did for me, is cache folder cleanup from time to time
This addon sets the advancedsettings.xml file to most optimal values if you select the right options (I tried them all)
Conclusion: sounds good, doesn't work

Please uninstall this addon as it is banned from this forum.

Although you have provided a detailed description (thanks), you forgot the single most important piece of information we need. The Debug Log Please provide this so we can check what the problem is. Make sure you capture the problem in the log.

FWIW, I have HTPC <-> Cat 6 Lan <-> Storage also, and no problems playing full bluray rips as well as UHD Bluray. Nothing is tweaked, and I don't even have those advancedsettings.xml entries.

RE: The ethernal caching issue - source too slow and other buffering and stuttering... - pa-pa-americano - 2021-10-12

(2021-10-12, 03:49)Karellen Wrote:
(2021-10-12, 03:43)pa-pa-americano Wrote: OpenWizard addon:
https://www.kodi-guide.com/how-to-fix-kodi-buffering/

Now that I tried to fix the things manually, I decided to use this addon, long story short, it has no effect on caching, the only good thing this addon did for me, is cache folder cleanup from time to time
This addon sets the advancedsettings.xml file to most optimal values if you select the right options (I tried them all)
Conclusion: sounds good, doesn't work
Please uninstall this addon as it is banned from this forum.

Although you have provided a detailed description (thanks), you forgot the single most important piece of information we need. The Debug Log Please provide this so we can check what the problem is. Make sure you capture the problem in the log.

FWIW, I have HTPC <-> Cat 6 Lan <-> Storage also, and no problems playing full bluray rips as well as UHD Bluray. Nothing is tweaked, and I don't even have those advancedsettings.xml entries.

Thank you for your reply,

Sorry I didn't know, I will uninstall this addon, I wasn't trying to promote it, just to test if I made a mistake in the setting file

It will take me a while to capture the log for every possible option for this file, I'll try to do this this week

Could you please try to tell me with what speed you transfer a single file from your storage to your kodi?

RE: The ethernal caching issue - source too slow and other buffering and stuttering... - Karellen - 2021-10-12

(2021-10-12, 03:58)pa-pa-americano Wrote: Sorry I didn't know, I will uninstall this addon, I wasn't trying to promote it, just to test if I made a mistake in the setting file

No problems. I figured you were testing looking for solutions.

(2021-10-12, 03:58)pa-pa-americano Wrote: It will take me a while to capture the log for every possible option for this file, I'll try to do this this week

I don't need anything complicated. Play the movie so that it stutters. Ensure that the Debug Log is running so the logging is captured, then upload the log. I don't need multiple scenarios, just that stuttering. Let it run for a couple of minutes.

(2021-10-12, 03:58)pa-pa-americano Wrote: Could you please try to tell me with what speed you transfer a single file from your storage to your kodi?

Sorry I have no idea. It all works so I have never needed to figure out throughput rates

RE: The ethernal caching issue - source too slow and other buffering and stuttering... - pa-pa-americano - 2021-10-12

(2021-10-12, 04:05)Karellen Wrote:
(2021-10-12, 03:58)pa-pa-americano Wrote: Sorry I didn't know, I will uninstall this addon, I wasn't trying to promote it, just to test if I made a mistake in the setting file
No problems. I figured you were testing looking for solutions.

(2021-10-12, 03:58)pa-pa-americano Wrote: It will take me a while to capture the log for every possible option for this file, I'll try to do this this week
I don't need anything complicated. Play the movie so that it stutters. Ensure that the Debug Log is running so the logging is captured, then upload the log. I don't need multiple scenarios, just that stuttering. Let it run for a couple of minutes.

(2021-10-12, 03:58)pa-pa-americano Wrote: Could you please try to tell me with what speed you transfer a single file from your storage to your kodi?
Sorry I have no idea. It all works so I have never needed to figure out throughput rates

I guess your server is faster than mine, I still don't have a decent RAID card that supports more than 8 SATA ports that's compatible with modern linux up to date, anyway, here the requested logs

Start the log slicing with the following command right before starting the video from the library:
$> tail -f ~/.kodi/temp/kodi.log > test-1.log

Case 1:
ahasobukog.kodi (paste)
- removed the file advancedsettings.xml completely
- tested file is a movie of 60G

Case 2:
oziyuxezuh.kodi (paste)
- content of advancedsettings.xml:
zotehojoxa (paste)
- log of ram usage every second:
ezagazaqoc (paste)
- same file tested as in case 1
- observation: ~300M of ram taken during playback (maybe it works the other way I thought? I need to set the full ram available in the file and Kodi will take up to 1/3 to cache anything?)
- expectation: takes up to 900M

Case 3:
otihadikan.kodi (paste)
- content of advancedsettings.xml:
tevukumuci (paste)
- log of disk usage every second:
uxaxoruvid (paste)
- same file tested as in case 1
- observation: disk doesn't fill during playback
- expectation: disk fills during playback

Conclusion of that test:
In both cases 1 and 2, the playback seems a bit faster than in the 3rd case, but that's maybe my advanced state of fatigue this morning...

Notice that those tests were done without pausing the playback
After those 3 tests, I ran one last, with the advancedsettings.xml from the case 2, play for 1 minute, pause 1 minute, unpause, get instantly a buffering
It goes more or less the same way for the 3 tests during a playback, video plays ~10 seconds, video buffers during 2-20 seconds, video resumes for ~10 seconds

I hope it's enough for now, let me know if you need anything else

Regards

PS: I upgraded Kodi+LE yesterday, I'm running those tests on Kodi 19.1 with LE10

RE: The ethernal caching issue - source too slow and other buffering and stuttering... - Karellen - 2021-10-12

(2021-10-12, 05:39)pa-pa-americano Wrote: I hope it's enough for now, let me know if you need anything else

No, not enough. I need the full logs. The snippets you provided are not enough.

RE: The ethernal caching issue - source too slow and other buffering and stuttering... - pa-pa-americano - 2021-10-12

(2021-10-12, 05:47)Karellen Wrote:
(2021-10-12, 05:39)pa-pa-americano Wrote: I hope it's enough for now, let me know if you need anything else
No, not enough. I need the full logs. The snippets you provided are not enough.

Oh by bad, I'll provide the logs from the start of the hardware, not from the start of the playback as I originally thought

RE: The ethernal caching issue - source too slow and other buffering and stuttering... - pa-pa-americano - 2021-10-12

(2021-10-12, 05:57)pa-pa-americano Wrote:
(2021-10-12, 05:47)Karellen Wrote:
(2021-10-12, 05:39)pa-pa-americano Wrote: I hope it's enough for now, let me know if you need anything else
No, not enough. I need the full logs. The snippets you provided are not enough.
Oh by bad, I'll provide the logs from the start of the hardware, not from the start of the playback as I originally thought

Logs on from boot, not cut, not modified, playback ~2mins

1: enetebafec.kodi (paste)
2: https://paste.kodi.tv/oxifolehaw.kodi