[Uatschitchun]

Hi there...

I've got the following question:

I have 2 main music paths (/media/music & /media/audioplays) which are all in one DB. The library has multiple sources for those 2 paths (Reggae, Pop, Soundtrack as examples for Music & Comedy, Audiobooks, Podcasts as examples for Audioplays).

I'd like to split up those 2 main paths within Kodi for to not have Music cluttered with Audioplays and vice versa.

I created nodes like this:

Code:

<?xml version='1.0' encoding='UTF-8'?>
<node order="20" type="filter">
       <label>Interpreten</label>
       <icon>DefaultMusicArtists.png</icon>
       <content>artists</content>
       <rule field="path" operator="contains">
               <value>audioplay</value>
       </rule>
</node>

Problem is, as this is a "filter" type, it creates a query on the DB which takes ages (multiple minutes) as it's a large DB (250MB).

Together with this, the above node results in duplicate artists with exactly the same albums underneath!?

I tried using folder type nodes, but don't know how to restrict them by path?!

Together with this, the standard nodes sometimes have 

Code:

visible="Library.HasContent(Music)"

or 

Code:

visible="Library.HasContent(Compilations)"

Question: what's it about with these "visible" flags? Where and when to use them? Can't find any info on it.

So, what would your approach be to split the audio DB into 2 without creating everlasting queries?

[black_eagle]

I tried using folder type nodes, but don't know how to restrict them by path?!

If you've added different sources to your library, you can use a folder type node to pull out individual sources but you need to know the sourceid from the database (you can get it by navigating manually to the source you want, and looking in Kodi's debug log).

For instance, I have added a source for stage musicals and all the albums for that are in a specific path.  To get that source in a folder type node, I do this.

xml:
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>

<node order="150" type="folder">

    <label>Stage Musicals</label>

    <icon>DefaultMusicSources.png</icon>

    <path>musicdb://sources/30/?sourceid=30</path>

</node>

That then takes me to all the artists in that source, and from there to albums.  The visibility condition that you were asking about is simply to ensure that nodes are not displayed if there is no content to fill them with.  E.g. The compilations or boxset nodes are not present if you don't have any compilations or boxsets.

You can also do it this way (which is easier and takes you to albums not artists)

xml:
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>

<node order="150" type="folder">

    <label>Stage Musicals</label>

    <icon>DefaultMusicSources.png</icon>

    <path>musicdb://albums/?source=StageMusicals</path>

</node>

I guess it all depends on how your stuff has been added.  If you've added a source of say, comedy and another of AudioBooks then you should be able to pull out the contents of those sources in the same way.  BTW, it's not the size of the db that's the problem with filtering, it's the way Kodi itself builds the filter(s).  I know work is ongoing though to change this to be done in the db, which should be much faster.  There is already a PR for the initial part of the changes.

[Uatschitchun]

Thx for the explanation of visibility and nodes based on sources!

Is it possible to create folder type nodes based on path? Or is it possible to have multiple sources within one node?

What I would need is a distinction by path, like I did in my example, in an acceptable speed or a node with multiple sources, like:

Code:

<path>musicdb://sources/30/? sourceid=30</path>
<path>musicdb://sources/40/?sourceid=40</path>

which would show all artists from both sources.

In the end I'd like to have my music library split up in 2 (don't want to switch profiles anymore!)
Together with that, having 1 library served and maintained by 1 kodi headless instance would allow using yatse with multiple clients based on one instance only, which minimize the hassle of keeping every client updated within yatse and music & audioplays accessible without switching profiles (which need different ports for yatse to recognize the different profiles).

I tried my example above (artists) with genre and it took more than 10 minutes, so I killed kodi and the corresponding query in mariadb.

What could be the cause for having duplicate entries with the node in my last post? It also contains entries that may not pass the path rule!?

[PatK]

Is it possible to create folder type nodes based on path? Or is it possible to have multiple sources within one node?

In the library node editor, add rule, title should bring up a requester that includes 'path' (browse for value) you can have multiple rules/paths in one node, plus multiple nodes (most of the rules are the same as smart playlists).

my music library split up in 2

Should do, create two nodes or use smart playlists and call them into the nodes. six of one half dozen of the other... nodes may offer different features. I haven';t gone down this rabbit hole far enough to tell you if there will be issues with Yaste and a headless Kodi (I don't think this is supported directly) Perhaps you can enlighten us?

genre and it took more than 10 minutes

Naw: nothing takes 10 minutes in Kodi outside of a full scrape. Either a scrambled library or stumbling on the scraper. Duplicates are almost always duel different paths to the same thing, clean-up should take care of this if the library is not compromised. I knew you didn't want to hear that. Check  the path for duplicates and see if they point to the same file but a different path, usual issue if 'sources' scanned while nested-sources in the same folder are also 'sources'.

Side note Kodi 19+ atm has issues with the library node editor.

[DaveBlake]

Is it possible to create folder type nodes based on path? Or is it possible to have multiple sources within one node?

What I would need is a distinction by path, like I did in my example, in an acceptable speed or a node with multiple sources, like:
<path>musicdb://sources/30/? sourceid=30</path>
<path>musicdb://sources/40/?sourceid=40</path>
which would show all artists from both sources.

No, a folder type node has one <path> entry only, which contains an encoded decription of the node with limited filtering options.
Yes <path> can specify a source/sourceid as an option, but only one.

Try a different approach. Music sources can have more than one folder path, and sources can over lap. So if you add a new source (in files) that combines the folders you want, then make a folder type node for that source.

[Uatschitchun]

Code:

2019-12-09 07:47:13.407 T:140487817414784   DEBUG: CGUIMediaWindow::GetDirectory (library://music/hoerspiele/genres.xml/)
2019-12-09 07:47:13.407 T:140487817414784   DEBUG:   ParentPath = [library://music/hoerspiele/]
2019-12-09 07:47:13.407 T:140487171651328   DEBUG: Thread waiting start, auto delete: false
2019-12-09 07:47:13.414 T:140487171651328   DEBUG: This query part contains a like, we will double backslash in the next field:  LIKE '
2019-12-09 07:47:13.414 T:140487171651328   DEBUG: GetGenresNav query: SELECT genre.* FROM genre JOIN song_genre ON song_genre.idGenre = genre.idGenre JOIN songview ON songview.idSong = song_genre.idSong JOIN song_artist ON song_artist.idSong = songview.idSong JOIN artistview ON
artistview.idArtist = song_artist.idArtist WHERE ((( (EXISTS (SELECT DISTINCT song_artist.idArtist FROM song_artist JOIN song ON song.idSong = song_artist.idSong JOIN path ON song.idpath = path.idpath WHERE song_artist.idArtist = artistview.idArtist AND path.strPath LIKE '/medi
a/8TB/Audio/Hoerspiele/%'))))) AND (genre.strGenre != '') GROUP BY genre.idGenre
2019-12-09 07:47:13.507 T:140487817414784   DEBUG: ------ Window Init (DialogBusy.xml) ------
2019-12-09 07:47:18.489 T:140485716051712   DEBUG: Thread JobWorker 140485716051712 terminating (autodelete)
2019-12-09 07:47:18.645 T:140487817414784   DEBUG: ------ Window Init (Pointer.xml) ------
2019-12-09 07:47:18.932 T:140487817414784   DEBUG: ------ Window Deinit (Pointer.xml) ------
2019-12-09 07:54:30.920 T:140487171651328   DEBUG: Thread waiting 140487171651328 terminating
2019-12-09 07:54:30.933 T:140487817414784   DEBUG: ------ Window Deinit (DialogBusy.xml) ------
2019-12-09 07:54:30.934 T:140487817414784   DEBUG: Saving fileitems [musicdb://genres/?xsp=%7b%22group%22%3a%7b%22mixed%22%3afalse%2c%22type%22%3a%22genres%22%7d%2c%22rules%22%3a%7b%22and%22%3a%5b%7b%22field%22%3a%22path%22%2c%22operator%22%3a%22startswith%22%2c%22value%22%3a%5b
%22%2fmedia%2f8TB%2fAudio%2fHoerspiele%2f%22%5d%7d%5d%7d%2c%22type%22%3a%22artists%22%7d]
2019-12-09 07:54:30.936 T:140487817414784   DEBUG:   -- items: 308, sort method: 1, ascending: true
2019-12-09 07:54:30.936 T:140487817414784   DEBUG: Custom album format = [%B]
2019-12-09 07:54:30.979 T:140487171651328   DEBUG: Previous line repeats 1 times.
2019-12-09 07:54:30.979 T:140487171651328   DEBUG: Thread BackgroundLoader start, auto delete: false
2019-12-09 07:54:31.046 T:140487171651328   DEBUG: Thread BackgroundLoader 140487171651328 terminating

This is library://music/hoerspiele/genres.xml/

Code:

<?xml version='1.0' encoding='UTF-8'?>
<node order="10" type="filter">
       <label>135</label>
       <icon>DefaultMusicGenres.png</icon>
       <content>artists</content>
       <group>genres</group>
       <rule field="path" operator="startswith">
               <value>/media/8TB/Audio/Hoerspiele/</value>
       </rule>
</node>

Starting at 7:47 and returning at 7:54. Ok, 7 minutes
The library is freshly scanned

[the_other_guy]

Is it possible to create folder type nodes based on path? Or is it possible to have multiple sources within one node?
parent node  mymusic

node <path>musicdb://sources/30/? sourceid=30</path>
node <path>musicdb://sources/40/?sourceid=40</path>
parent node  mymusic
parent node Music
node
node
parent node Comedy,
node
node

<?xml version='1.0' encoding='UTF-8'?>
<node type="filter">
    <label>artist</label>
    <content>artists</content>
    <rule field="artist" operator="is">
        <value>Adam Harvey</value>
    </rule>
    <rule field="artist" operator="is">
        <value>Beccy Cole</value>
    </rule>
    <match>any</match>
    <rule field="artist" operator="is">
        <value>Graeme Connors</value>
    </rule>
</node>


Side note Kodi 19+ atm has issues with the library node editor.
python 3 has bits removed or replaced and will need a workaround 
from htmlentitydefs import name2codepoint
The htmlentitydefs module has been renamed to html.entities in Python 3.
from urlparse import parse_qs 
The urlparse module is renamed to urllib.parse in Python 3

[Uatschitchun]

Ok, what works is the following:

If I have the following sources added within "files"

• /media/audioplay/comedy => As "comedy" (Sourceid=1)
  
• /media/audioplay/audiobooks => As "audiobooks" (Sourceid=2)
  
• /media/music/pop => As "pop" (Sourceid=3)
  
• /media/music/rock => As "rock" (Sourceid=4)
  

Each as individual source, those 4 sources appear in music->sources node. So far so good..

If I add another 2 (combined) sources within "files":

• /media/audioplay/comedy & /media/audioplay/audiobooks => As "Audioplays" (Sourceid=5)
  
• /media/music/pop => As "pop" & /media/music/rock => As "Music" (Sourceid=6)
  

I have a total of 6 sources.

The standard music nodes give "Artists", "Albums", "Genre", "Recentlyadded", etc. for all sources.

This one gives "Artists" for "comedy"

Code:

<?xml version='1.0' encoding='UTF-8'?>
<node order="20" type="folder">
       <label>Artists Comedy</label>
       <icon>DefaultMusicGenres.png</icon>
       <path>musicdb://artists/?sourceid=1</path>
</node>

and this one for "Audioplays"

Code:

<?xml version='1.0' encoding='UTF-8'?>
<node order="20" type="folder">
       <label>Artists Audioplays</label>
       <icon>DefaultMusicGenres.png</icon>
       <path>musicdb://artists/?sourceid=5</path>
</node>

Same works for "Albums" by using

Code:

<path>musicdb://albums/?sourceid=5</path>

So with this one's able to separate "Music" from "Audioplays" for artists, albums & songs

Not working is this one:

Code:

<path>musicdb://genres/?sourceid=5</path>

it delivers all existent genres where the ones not appearing in "sourceid=5" (Audioplays) are simply empty ;(

Also not able to filter by sourceid are: 

• musicdb://recentlyaddedalbums/ (using musicdb://recentlyaddedalbums/?sourceid=5 is equal to using musicdb://recentlyaddedalbums/)
  
• musicdb://years/ (it delivers the same result as musicdb://genres/ filtered by sourceid => a list with all years in DB with empty entries)
  

I've searched, but could not find any documentation (besides this one: https://kodi.wiki/view/Opening_Windows_a...logs#Music) in regards to folder type nodes using musicdb:// paths and the filtering options...

[the_other_guy]

<path>musicdb://genres/?sourceid=5</path>  change sourceid to genreid
open genres add to favourites open C:\Users\*****\AppData\Roaming\Kodi\userdata\favourites.xml
<favourite name="Comedy">ActivateWindow(10502,&quot;musicdb://genres/4/&quot;,return)</favourite>

musicdb://genres/4
<?xml version='1.0' encoding='UTF-8'?>
<node type="folder">
    <label>test</label>
    <path>musicdb://genres/?genreid=4</path>
</node>

you can also set path on
 <?xml version='1.0' encoding='UTF-8'?>
<node type="filter">
    <label>test</label>
    <content>albums</content>
    <rule field="path" operator="contains">
        <value>E:\music\Beccy Cole\</value>
    </rule>
    <order direction="ascending">year</order>
</node>

[Uatschitchun]

First, thx @the_other_guy , but I have to tell that your posts often are hard to understand

<path>musicdb://genres/?sourceid=5</path>  change sourceid to genreid
open genres add to favourites open C:\Users\*****\AppData\Roaming\Kodi\userdata\favourites.xml
<favourite name="Comedy">ActivateWindow(10502,&quot;musicdb://genres/4/&quot;,return)</favourite>

musicdb://genres/4
<?xml version='1.0' encoding='UTF-8'?>
<node type="folder">
    <label>test</label>
    <path>musicdb://genres/?genreid=4</path>
</node>

Yes, that would help show a specific genre, but I would like to have all genres for a specific path or source. A node like this:

Code:

<?xml version='1.0' encoding='UTF-8'?>
<node type="filter">
<label>Genre-test</label>
<content>artists</content>
<group>genres</group>
<rule field="path" operator="contains">
<value>/media/audioplay/</value></rule>
</node>

Takes 7 minutes and it results in a mostly empty genres not appearing on /media/audioplay/ ;(
Even running the query on the DB-host itself (MariaDB, core i5-8400, SSD) gives:

Code:

MariaDB [headlesstest-audio72]> SELECT genre.* FROM genre JOIN song_genre ON song_genre.idGenre = genre.idGenre JOIN songview ON songview.idSong = song_genre.idSong JOIN song_artist ON song_artist.idSong = songview.idSong JOIN artistview ON artistv
iew.idArtist = song_artist.idArtist WHERE ((( (EXISTS (SELECT DISTINCT song_artist.idArtist FROM song_artist JOIN song ON song.idSong = song_artist.idSong JOIN path ON song.idpath = path.idpath WHERE song_artist.idArtist = artistview.idArtist AND p
ath.strPath LIKE '%/media/audioplay/%'))))) AND (genre.strGenre != '') GROUP BY genre.idGenre;
....
308 rows in set (7 min 33.934 sec)

[DaveBlake]

Yes, that would help show a specific genre, but I would like to have all genres for a specific path or source.

Frankly, the SQL produced at the moment for this node rule combination (with group) is pretty inefficient, and on your huge music library is just going to choke (like you report). I have WIP to deal with building these queries better, but it is a big job and I keep getting pulled onto other things. However as a data guy it is something dear to my heart so I hope to get there eventually.

Most users have much smaller libraries, and much simpler filtering requirements, they will see your 7mins as an error. Hope my sources suggestion was useful.

[Uatschitchun]

Yes, sources suggestion was helpful indeed, as it allows folder type node with using the sourceid of a combined source.
One question here. If my sources are for instance
/media/audioplay/comedy
/media/audioplay/audiobooks
is it ok to add a single path node
/media/audioplay/
or should it better be a multipath source of the single sources above?

Would adding the sourceid filter option for musicdb://years,genres and recentlies speed up things?

I thought that it's about choking in regards to the genre query filtered by path. Together with the slowness, it creates wrong (empty) results.

If you'd like me to test, you know how to reach me!

Conclusion until now:
Filter type queries on genre by path are slow as hell
Folder type queries work for every item that allows sourceid filtering (artists, albums). A list of possible filter options for folder type queries (musicdb://...) would be glorious @Dave!

[DaveBlake]

If my sources are for instance
/media/audioplay/comedy
/media/audioplay/audiobooks
is it ok to add a single path node
/media/audioplay/
or should it better be a multipath source of the single sources above?

Having a music source with a single parent folder, and then other music sources with subfolders is fine.
 

Would adding the sourceid filter option for musicdb://years,genres and recentlies speed up things?

It is still about me redesigning the way SQL queries are built for all the filtering variations, more than how "fetch me the genres for songs on this source" is specified using filter type node (with rules and group) or folder type node (with db url and options).

 

Filter type queries on genre by path are slow as hell

Filter type nodes on artists with source or path or genre rules that are grouped by genre are seriously flawed. It is the <group>genres</group> that causes some very ugly SQL to be generated when combined with rules that do not apply directly to fields of the artist table e.g. source, path and genre all mean looking at the songs/albums of the artists.

 

Folder type queries work for every item that allows sourceid filtering (artists, albums). A list of possible filter options for folder type queries (musicdb://...) would be glorious @Dave!

Folder type nodes is an undocumented feature
It takes advantage of how screen navigation historically works internally, but I think the long term plan was to remove that type of node entirely. Using the options on folder type nodes for filtering was useful accident rather than design. However don't worry, not going to happen until the filter type nodes (which do offer much greater filtering possibilites) can support all the navigation behaviour that folder type nodes offer. My work is to achieve that and then there will be no need for users to use folder type nodes, they will be able to get what they want with the much clearer filter type.

[TimoJ]

Therefore path for to Movies node is library:/videos/movies and to explicitly go to tiles it is library:/videos/movies/titles.xml

I have changed Confluence skin's Movies homebutton (with home.xml editing) to point to a custom version of library:/videos/movies/titles.xml and it works fine. But there is a problem: if I go to the listing's top row and press those two dots, the listing doesn't show my other custom nodes, instead it shows just default database selections. Where is this path coming from? From the Confluence skin or somewhere else? Still, if I go down one level more, there is again Movies and my custom nodes are in there and inside there the two dots navigation works fine.

[jjd-uk]

That sounds like a bug.