[jeffski10]

Its path substitution - do this for sources, playlists and favourites to keep all devices using the same settings.

[BatterPudding]

Its path substitution - do this for sources, playlists and favourites to keep all devices using the same settings.

That makes sense. For now you should be able to manually specify where your sources.xml is using the settings page.

This and other tweaks are currently being discussed via carrier pigeon now.

[black_eagle]

Its path substitution - do this for sources, playlists and favourites to keep all devices using the same settings.

It is not necessary to path sub sources.xml with a MySQL db, as all the paths in it are absolute network paths. The only reason you might want to do this is if you use more than one machine to scan to your library with. I have 5 kodi instances using a shared database, but only one sources.xml on the machine that updates the db.

I will however look into the possibility that we can pull the path subs from AS.xml and use those if they exist. Of course, this still doesn't work for people like me who don't substitute the sources file so the add-on would still need to retain the option to set that manually.

[MacGyver]

So, that just JACKED my database. It was a huge 30mb MySQL database that has been updated from release to release. It removed lots of stuff, 4/5 of the movies database, and it confused the tv database (it says it has 19 things in one show, yet nothing displays).
Does your script work correctly with sources that have spaces in the name by chance? How about multi-source sources?

Code:

<source>
            <name>All Movies</name>
            <path pathversion="1">smb://192.168.0.103/Media/Movies/Action/</path>
            <path pathversion="1">smb://192.168.0.103/Media/Movies/Comedy/</path>
            <path pathversion="1">smb://192.168.0.103/Media/Movies/Drama/</path>
            <path pathversion="1">smb://192.168.0.103/Media/Movies/Horror/</path>
            <path pathversion="1">smb://192.168.0.103/Media/Movies/Sci-Fi/</path>
            <path pathversion="1">smb://192.168.0.103/Media/Movies/Family/</path>
            <path pathversion="1">smb://192.168.0.103/Media/Movies/Kid Action/</path>
            <path pathversion="1">smb://192.168.0.103/Media/Movies/Kid Comedy/</path>
            <path pathversion="1">smb://192.168.0.103/Media/Movies/Kid Drama/</path>
            <path pathversion="1">smb://192.168.0.103/Media/Movies/Kid Horror/</path>
            <path pathversion="1">smb://192.168.0.103/Media/Movies/Kid Sci-Fi/</path>
            <thumbnail pathversion="1">/storage/.xbmc/userdata/Icons/all.png</thumbnail>
            <allowsharing>true</allowsharing>
        </source>

Sources like that?

I have a backup so I'm cool, but you need to check some things.

Thanks for trying to create a tool to clean up the database. Let me know when 0.6.0 is available.

[black_eagle]

With regard to sources with spaces in, this is an excerpt from my sources.xml which I have tested against multiple times, so I'm 99.9% certain spaces are not an issue.

Code:

<source>
            <name>Kids Films (2)</name>
            <path pathversion="1">smb://HS-DHGL925/share/Kids Films/</path>
            <allowsharing>true</allowsharing>
        </source>
        <source>
            <name>More TV</name>
            <path pathversion="1">/media/nas/More TV/</path>
            <allowsharing>true</allowsharing>
        </source>
        <source>
            <name>More Movies</name>
            <path pathversion="1">/media/nas/More Movies/</path>
            <allowsharing>true</allowsharing>
        </source>

This is against local and network shares as you can see.

I was not aware of multi-source sources such as in your example, so I will have to do some checking there.

Sorry about your database, but that is the point of a beta test, to uncover any issues that we may have missed in our initial testing. This is why we strongly encourage users to back up their databases first before running the script· Fortunately you have done so which I am relieved to hear. Did you by any chance have debugging turned on in both Kodi and the script settings ? I'd be interested to see the relevant part of the log if so, particularly the SQL line the script produced.

[BatterPudding]

My testing also kicked multiple sources around, not just with spaces but also some Chinese text in there too.

Interesting to see the different layout of your sources.xml. Is that hand written as the KODI produced one looks more like what black_eagle pasted above?

[MacGyver]

It is hand written but just expanded from a gui created one from years ago. Like 9 years ago. And that was one of 12 or so multi-source sources.
Also I blend multi-source sources that can contain the same folders ie. "SNL" in one share and an "SNL" in another, each containing different files. Kodi merges the two on one device but other devices have access to only one half.
Sorry I did not have the log turned on.
Afterwards the database had weird things going on in Tv Shows, lots of things had show totals, but when opening them there were just ".." and no files, and like I said about 4/5 of the movies just disappeared.
I did do a reboot, and a Kodi database cleaning, but the weirdness persisted, so I restored my backup.
My database is MySQL version 5.6.29 running on Mint Linux 17.2.

I can say without a doubt it did "clean" out the library.

But, seriously, let me know when you want me to try a new version. I have a crazy database so no doubt you couldn't plan for for all the craziness.

[black_eagle]

Thanks for all the feedback, it is invaluable.

Got something in-place to handle multiple paths for a single source although I'm not sure what might happen with your 'SNL' scenario. It should be excluded from the clean as everything found in the sources.xml is automatically excluded (or it should be !!).

Going to try and push out an updated version later tonight. Still working on the code but it should be ready soon-ish. Changelog will detail what was changed/fixed.

Just please, back up first !!

[BatterPudding]

Thanks MacGyver, you are the exact victim willing test subject we created this thread for. Thanks for your help and the details. KODI documents so much in the wiki, and we follow what we can, but KODI is also insanely flexible. So KODI is reading in your multi-paths source and happily working with it, even though it isn't documented.

I would say that what happened to you was we read the first path from that list of sources, and ignored the rest. That would then explain why it did such an efficient Cleaning Job.

We did say it was a junk cleaner...


Now I'm just waiting for Little Bobby Tables to turn up ( https://www.xkcd.com/327/ )

[MacGyver]

Oh, and I have LOTS of merged files from split files ie. It (1984) (1of4).avi, It (1984) (2of4).avi, etc... all showing up as one singular library entry "It (1984)" in the library by using a regex in advancesettings.xml like:

Code:

<regexp>(.*?)([ \(]+[1-9]+of+[1-9]+[\)])(.*?)(\.[^.]+)$</regexp>

Let me know if you need anything else form me. I probably have all the bad database problems anybody could have. (hence me trying out a database cleaning tool)


Thanks for all the work so far by the way.

[BatterPudding]

Oh, and I have LOTS of merged files from split files ie. It (1984) (1of4).avi, It (1984) (2of4).avi, etc... all showing up as one singular library entry "It (1984)" in the library by using a regex in advancesettings.xml like:

Code:

<regexp>(.*?)([ \(]+[1-9]+of+[1-9]+[\)])(.*?)(\.[^.]+)$</regexp>

That should be okay. We are hunting down files that are not in a "keep it" path list. So as long as all of those parts lurk in a "keep it" under folder path, then that will be fine.

Your film could be in 50 different parts, scattered between your different source folders, but as long as those files are in folders listed in sources.xml, then all will be good.

[black_eagle]

We have just pushed an update to the repo. BatterPudding I believe is posting the changelog in the second post of this thread so we can keep you all updated with events via the first page.

For clarity, the changes are as follows.

I've added a fix for @MacGyver's fancy sources.xml which I have tested but back-up before using is clearly still the order of the day.

Removed is the need to click on OK when the script tells you it's running the built in clean library routine. This has been changed to a 5 second notification to allow the script to run silently for those users who wish to run it on a schedule.

That's it. Happy cleaning

[orby]

just tried running it and it removed all of my movies, sources.xml below, backup restored and all is good again thank you for your time and work looking forward to blowing something up again in the near future hehe

Code:

<video>
        <default pathversion="1"></default>
        <source>
            <name>TVShows</name>
            <path pathversion="1">nfs://192.168.1.7/Movies/TVShows/</path>
            <allowsharing>true</allowsharing>
        </source>
        <source>
            <name>movies1</name>
            <path pathversion="1">W:\</path>
            <allowsharing>true</allowsharing>
        </source>
        <source>
            <name>movies2</name>
            <path pathversion="1">V:\</path>
            <allowsharing>true</allowsharing>
        </source>
        <source>
            <name>movies3</name>
            <path pathversion="1">T:\</path>
            <allowsharing>true</allowsharing>
        </source>
        <source>
            <name>movies4</name>
            <path pathversion="1">S:\</path>
            <allowsharing>true</allowsharing>
        </source>
        <source>
            <name>movies6</name>
            <path pathversion="1">P:\</path>
            <allowsharing>true</allowsharing>
        </source>
        <source>
            <name>movies7</name>
            <path pathversion="1">O:\</path>
            <allowsharing>true</allowsharing>
        </source>
        <source>
            <name>movies8</name>
            <path pathversion="1">X:\</path>
            <allowsharing>true</allowsharing>
        </source>
    </video>

[MacGyver]

I just tried 0.5.1 and it worked great, and the built-in kodi clean is now 10x faster. I haven't noticed any weird side effects, so far it looks good here.

Thanks guys. The built-in kodi speed up alone was worth the first restore.

I'll do a more thorough look when I get off work tomorrow, but it seems to have worked perfect this time.

Thanks again guys.

[black_eagle]

just tried running it and it removed all of my movies, sources.xml below, backup restored and all is good again thank you for your time and work looking forward to blowing something up again in the near future hehe

I know it might be a pain, but can you trash your database again with debugging turned on in both Kodi and the add-on settings and post the relevant part of the debug log please. At first glance your sources.xml doesn't look as if it has any issues in it.

Cheers.

I just tried 0.5.1 and it worked great, and the built-in kodi clean is now 10x faster. I haven't noticed any weird side effects, so far it looks good here.

Thanks guys. The built-in kodi speed up alone was worth the first restore.

I'll do a more thorough look when I get off work tomorrow, but it seems to have worked perfect this time.

Thanks again guys.

Good to hear that the fix worked and that you don't appear to have any issues with your db and thanks for the feedback