[DaveBlake]

I have just discovered the consequences for my music library of clicking on Album Information for a single album. I knew that there was an option to query information for all albums, and that that used scrapers to get info from online databases etc. it turns out that the scraper is also used for a single album.

Why is this an issue? Well when the scraper gets it wrong it quietly removes the correct album artist data that had been scanned into the library from tags. So for example I choose album information on the context menu for my Dvorak Symphony no. 7 by Jarvi/SNO album on Chandos, it finds a Colin Davis/LSO on a different label recording and adds them as artists but deletes Jarvi and SNO. One tiny click, no promting or warnings, and they are gone!!

The album and songs are still there and show the original artist string when playing or in lists so it took me a while to notice, but the song_artist and album_artist tables entries have been cleared. Since it is the only album by that conductor and orchestra "Jarvi" and "SNO" do not appear in the artists list and Kodi doesn't find the album or songs when I search for those names.

I can see in the code why this happens. I understand that scrapers can be useful and powerful things, and that correctly identifying recordings can be difficult. What I question is if this quiet corruption of the music library is the behaviour we really want?

I think that at very least initiating scraping should be a separate user choice from looking at information. I knew I did not want to scrape, or if I did I wanted to see and confirm what updates happen. Maybe you want an auto scrape of the entire library, fine have that as an option but unbidden deletion of correct data is not good behaviour.

Thoughts anyone?

[scott967]

I'm guessing what you see is based on the "last scraped" attribute for the album. If it is null I think that triggers the scraper when you launch the album info dialog. My work-around has been that after I add/update the music library I execute an SQL command that fills in a dummy date for any null last scraped date (both in albums and artists tables). Another thing you can do, is after executing the SQL do a library export to separate files (it won't export if there is null "last scraped") so you have album.nfo file for it for the future (or to manually edit the data).

I'm not sure what the value add is of the check on "last scraped date" myself. It just seems to cause me more work. I think it would be sufficient to always just open the album info dialog, and let the user launch the scraper with the "reload" button if desired. A related problem is if the scraper doesn't find anything you can never get the album info dialog to open, you are stuck (but at least you don't get unwanted data)

scott s.
.

[DaveBlake]

Thanks for the SQL based work arounds Scott.

I'm not sure what the value add is of the check on "last scraped date" myself. It just seems to cause me more work. I think it would be sufficient to always just open the album info dialog, and let the user launch the scraper with the "reload" button if desired. A related problem is if the scraper doesn't find anything you can never get the album info dialog to open, you are stuck (but at least you don't get unwanted data)

I think that this aspect of the music library could benefit from some attention. Scraping is all well and good, but user control of when scraping happens and that unfriendly infinite loop when nothing is found needs fixing.

I am looking at other aspects of the music library, so I will put that on my list too. Anyone else care to comment on what behaviour would be most useful?

[DaveBlake]

Just updating this in case anyone else is interested. There is a discussion between devs (now locked to collaborators only) on changing what dialogs/menu items the user sees when adding a music source 8069 (PR).

The initial suggestion is to make music look more like video, move the choice of scraper nearer the file list (rather than in system settings). However there are issues and differences between video - where the are different kinds of content e.g. TV, Season, Movie etc., all data is scraped from online sources, and music - where curently there is only one kind of content (music, although could add audio books), the fundamental data is in the files themselves available off line, and scraping externally provides additonal information. There are also benefits to being able to scrap albums and artists separately.

Some contributors seem to believe that scraped data does not override that collected from the tags, but in my experience it does. It does not change the music files themselves, but it can delete artists from the database and replace them with others. In my case incorrectly too, because the scraper mis-identified the recording. I would reitterate that on the PR but I can't because I am locked out!

Anyone else had unwanted scraping experiences?

[WelshPaul]

I actually reported the reverse behaviour some time ago.
http://forum.kodi.tv/showthread.php?tid=215665

[jjd-uk]

This doesn't appear to be clearly documented anywhere, but my belief is that nfo's are only used to fill in data not available in the tags, so if there is a tag available for a certain field then the tag data will always be used.

[jjd-uk]

Some contributors seem to believe that scraped data does not override that collected from the tags, but in my experience it does. It does not change the music files themselves, but it can delete artists from the database and replace them with others. In my case incorrectly too, because the scraper mis-identified the recording. I would reitterate that on the PR but I can't because I am locked out!

Anyone else had unwanted scraping experiences?

I've not seen this personally, but I thought the only instance in which the stored metadata in the db gets changed is if you're using MB ID's and have the setting "Override song tags with online information" on.

[DaveBlake]

I actually reported the reverse behaviour some time ago.
http://forum.kodi.tv/showthread.php?tid=215665

If rescanning (updating the library) reapplied the tag data, (rightly so I think, I love my tag data), could you not scrape again for the NFO data?

But your issue might be one of data design too. Songs, albums and artists all have genre data stored in the library, but only the song data is used in playlists etc. The artist and album genre values are displayed as information but not queried. Kodi looks at the artists and albums with songs of genre "x" and shows you those. I am happy about it since when scraping artist info I get all kinds of genres stored for the artists thankfully none of it is subsequently used. But it is thwarting your attempts to use NFO to separate duplicates.

I think we can agree that the effects of scanning and scraping are not well described, and that user requirements are diverse!

[WelshPaul]

If rescanning (updating the library) reapplied the tag data, (rightly so I think, I love my tag data), could you not scrape again for the NFO data?

It doesn't seems to work. If I go to Album Information and re-load data it doesn't reload the nfo genre. A correction to my earlier post - it seems to be the final track that's being used. So for example I have 'It Takes a Nation of Millions' by Public Enemy, which includes the track 'Bring Tha Noize'. I also have Apocalypse '91 which has as the final track 'Bring Tha Noize (feat Anthrax)'. Now I'm a bit of a traditionalist when it comes to my hip-hop so I mark the former as genre = hip hop and the latter as genre = duplicate, as I'm not sure Anthrax are bringing that much to the party. In my nfo's both albums have the tags <genre>Hip Hop</genre>. However, after scanning to library Apocalypse '91 will be shown as Genre = Duplicate. The only way I can remove this is to edit the actual Album table in the database, though re-scanning will bring back the erronerous 'duplicate'. Hence I never re-scan my whole library nor do I ever export my library.

This doesn't appear to be clearly documented anywhere, but my belief is that nfo's are only used to fill in data not available in the tags, so if there is a tag available for a certain field then the tag data will always be used.

Hmm, you may be right but it doesn't ring true for our other contentious issue - ratings. So Beyoncé generally has one good track per album (unless other family members have been doing the tagging). So while an individual track may get a 5 out of 5 (or 10 out of 10) rating the album as a whole may be a 2 or 3 (out of something). In this case Kodi does seem to be picking up the album.nfo rating tag rather than the individual track rating tag. So, it's all a bit confusing.

[DaveBlake]

In a sense song genre tag data takes presidence over album genre NFO or scraper provided data. It is not as universally simple as tags over scraped, but in this instance, as you have discovered Paul, your approach does not work. To be honset I would not put tag data in my music files that I did not expect to see in Kodi, in fact there is lots of tag data I want to see that Kodi doesn't (yet) process. But yes it is confusing.

"Duplicate" is a genre of a song in the Apocalypse '91 album, so when you view albums for genre "Duplicate" you will see Apocalypse '91 in the list. That is how those nodes work, they use the song data. But having loaded the NFO when you look at album info for Apocalypse '91 it will say "Hip Hop".

This doesn't appear to be clearly documented anywhere, but my belief is that nfo's are only used to fill in data not available in the tags, so if there is a tag available for a certain field then the tag data will always be used.

The original idea with NFOs (and scrapers) seems to be to provide additional data that could not be gleened from tags in the music files. There is some attempt at merging too, depending on your settings, I have seen the code, and sometimes user provided data is not used in the way users might expect. But having had accidentally scraped and had Neeme Jarvi / SNO (the artists in the tag data) replaced by Colin Davies / LSO I know that scraping can override tags!

[WelshPaul]

"Duplicate" is a genre of a song in the Apocalypse '91 album, so when you view albums for genre "Duplicate" you will see Apocalypse '91 in the list. That is how those nodes work, they use the song data. But having loaded the NFO when you look at album info for Apocalypse '91 it will say "Hip Hop".

No it doesn't, having loaded the nfo it says Duplicate, not Hip Hop - that's the point, therefore the album won't show in playlist using hip hop genre as a criteria. I don't necessarily buy into the Kodi is an eco-system like Apple argument. I think many people still use other programs, I for one still haven't found anything better than Winamp and Milkdrop for visualization, and that hasn't been maintained for like a decade - now that's the concord or space shuttle of music players. Therefore, I really don't like the argument that we should tag everything to fit with the way Kodi does it. Though I do accept that will mean not everything works precisely as we would each like it to. So, I'm not saying it's all broke, just pointing out a minor glitch.

[DaveBlake]

No it doesn't, having loaded the nfo it says Duplicate, not Hip Hop

Where does it say "Duplicate"? Do you not see "Hip Hop" in when you show album info from the context menu? Not in a position to test out what you are doing for myself, and just curious to understand your glitch!

..the album won't show in playlist using hip hop genre as a criteria.

No it won't. Like I said Kodi does not use genre stored with album (the one NFO sets) in playlists, it uses the genre stored with songs (from tags) even if it is listing albums. Confusing perhaps, but comes from the days when the only data came from song file tags.

...Therefore, I really don't like the argument that we should tag everything to fit with the way Kodi does it. Though I do accept that will mean not everything works precisely as we would each like it to. So, I'm not saying it's all broke, just pointing out a minor glitch.

No worries, not wanting to force you into anything. Out of interest what does the space shuttle Winamp do with a song tagged with a genre of "duplcate"?

[WelshPaul]

As shown above it's in the Album Info screen.

Concord (maybe with an 'e' at the end??) and the space shuttle going out of service without an improved replacement are examples of technlogy going backwards, which is kind of how I feel about Winamp never really having been bettered, in spite of it's abuse at the hands of AOL. N.B. and why we should treasure Kodi and those who keep it safe.

I'm sorry, didn't mean to hijack the thread, just getting hyper, what with the rugby world cup starting tonight and all.

[Ned Scott]

Just updating this in case anyone else is interested. There is a discussion between devs (now locked to collaborators only) on changing what dialogs/menu items the user sees when adding a music source https://github.com/xbmc/xbmc/pull/8069.

Honest question, why was it locked? It doesn't seem like a heated discussion, the posts were on-topic, and locking it prevents people like notspiff (long time core dev who knows this stuff better than most) from commenting.

[DaveBlake]

Well that wasn't what I expected! Going to have to try it out now when I get a chance.

Technology does go backwards, and I feel sad/mad about Concorde (yes there's an "e"). But never used Winamp, it is RPi that made me take the CD-less plunge. I encouraged the conversation so no hijack. Enjoy the rugby!

Meanwhile anyone else got a scraper/album info/NFO story?