[DaveBlake]

You have been busy, good
So Adam Ant is a scraper issue, not tags. That is a relief actually, because it was driving me crazy to see how the tagging was doing it. I'm not so on the ball with scraping as I am with tags but I need to learn, so let us voyage together.

I have to say I would not let the scraper override my library data that is derived from tag values. I tag with correct artists etc., I really don't want that changed. Sure fetch artwork and biogs for artists, or reviews for albums and other peoples idea of theme, mood, rating etc., but don't change the artists, date, genre etc. There was a bug in Isengard that meant even with the "override tag" option diasbled it still changed the artists, which was a disaster when the scraper wrongly identified classical recordings. That is now fixed. My recommendation is scan to library with "fetch extra data on update" disabled, and be happy your tags have produced the results you want, then scrape artwork etc. but with "override tags" disabled. Keep control of your library, you are experiencing the kinds of things that can happen.

Originally, when Adam first appeared I believe that extra info was indeed disabled, but then I turned it on hoping it would correct things.

Na, I suspect that scraping happened. If it is the cause now of Adam Ant filling your library, then it was the cause then made worse by artist name to mbid count mis-match. Although the idea of scraping is that the user can get away without tagging, and Kodi just magically finds the data the reality is that for now you need accurate tags to enable Kodi to identify what needs to be scraped. One day it could be all audio fingerprint based, but not just yet. Meanwhile it seems there is some scraper bug too that is wreaking havoc in your library.

A bit odd to have Leona Lewis "Spirit" album in their twice, and without mbid if you tagged with Picard.

The MySQL stuff is indeed all new to me. I've used phpMyAdmin for looking through stuff, but that's about it. Not done any command line queries before. Even setting the bCompilation flag had me worried in case it did something else !!!

SQL is lovely, and sticking to "SELECT" statements perfectly safe. Just take a copy of the database if you are feeling insecure.

It is sounding less like m4a is contributing, but we will keep an eye out.

The scraping of artist info for Sinéad O'Connor using Universal scraper........ What is happening there? I don't know about scraper using cache, I think it clears it. I'll create a fake Capture album in m4a and try it myself. Money is on the ' or ` on O'Connor causing an issue But why Adam Ant?

You didn't report if you had him in the library with his own mbid (check artist table, and song_artist for the songs)

[DaveBlake]

Reading that debug extract it looks like you have an NFO file with Adam Ant in it somewhere. Do you? Search that drive!

What folder hierarchy are you using for those compilation albums like Capture?
What are your scraper settings?

The artist scraper trys to find a common path for an artist, and looks there for NFO. It could be looking somewhere unexpected.

I scraped Sinéad O'Connor, that was just her, no problem but that was with my settings and folder layout, and no NFO anywhere.

[black_eagle]

Reading that debug extract it looks like you have an NFO file with Adam Ant in it somewhere. Do you? Search that drive!

What folder hierarchy are you using for those compilation albums like Capture?
What are your scraper settings?

The artist scraper trys to find a common path for an artist, and looks there for NFO. It could be looking somewhere unexpected.

I scraped Sinéad O'Connor, that was just her, no problem but that was with my settings and folder layout, and no NFO anywhere.

No, I don't have an NFO with Adam Ant in it !! I used to have, because I have a 'Greatest Hits' anthology of 'Adam & the Ants' but when I first noticed this issue I'm having, I moved that album out of the music path, dropped the DB and scanned again. When he appeared again, I deleted the NFO file only to find it didn't help. That's when I thought of caching. However, I've just been looking at the Scraper.cpp code and although it does create a scraper cache the contents are deleted based on age so now I don't see how it could be that. Besides I've checked what I think is the right path '.kodi/temp/scrapers/metadata.artists.universal/' and the only items in there relate to Motorhead.

Scraper is V3.6.2 of the Universal Artist Scraper and all settings are at their default. Under Audio/Library, the only thing 'on' is 'Fetch additional Information during updates' - scrapers are set to Universal Album & Artist scrapers. All other settings are 'off'.

Directory layout is a top level directory containing directories each named with an artist. Each artist directory contains one or more sub-directories named as the relevant album. Each album directory contains the relevant music tracks, associated art, logos etc.

Currently, I have changed the root 'Music' path to only include a handful of artists and albums for testing purposes. None of the albums in this path have either artist or album.nfo's.....

[DaveBlake]

No, I don't have an NFO with Adam Ant in it !!

Sorry, but you know I had to ask.

I used to have, because I have a 'Greatest Hits' anthology of 'Adam & the Ants' but when I first noticed this issue I'm having, I moved that album out of the music path, dropped the DB and scanned again. When he appeared again, I deleted the NFO file only to find it didn't help. That's when I thought of caching. However, I've just been looking at the Scraper.cpp code and although it does create a scraper cache the contents are deleted based on age so now I don't see how it could be that. Besides I've checked what I think is the right path '.kodi/temp/scrapers/metadata.artists.universal/' and the only items in there relate to Motorhead.

Hummm, Motorhead.... They have had the Adam Ant treatment from your album_artist query results. Not sure what that may mean though, Adam still hiding in the cache?

This whole thing is quite bonkers. If I can repeat it then I can pin it down, but of course every thing works as it should when I am looking.

Directory layout is a top level directory containing directories each named with an artist. Each artist directory contains one or more sub-directories named as the relevant album. Each album directory contains the relevant music tracks, associated art, logos etc.

Currently, I have changed the root 'Music' path to only include a handful of artists and albums for testing purposes. None of the albums in this path have either artist or album.nfo's.....

So the Capture tracks are in something like .../Afro Celt Sound System/Capture 1995-2010/07Mother.m4a etc.?

[black_eagle]

Yes I know you had to ask. It is after all, the most logical explanation.

Built Isengard from source earlier today and installed and ran it in portable mode, just scanning those few albums. Nothing in the debug log for Adam at all although plenty of 503:Service unavailable errors from Musicbrainz.

Currently I'm scanning inside every file on the PC for "Adam Ant" to see if this finds anything. It's going to take a while though

Yeah, your directory structure is spot on. I don't see that any sort of share could be causing this but for what it's worth, the music directory is an NFS share. The full path for that album is NFS://media/sde1/Music/Afro Celt Sound System/Capture 1995-2010/1-01 Lagan.m4a etc.

Kodi is pointed at NFS://media/sde1/Music/ as the root directory. Now it did occur to me that there are other NFS exports as well, and that that might be where something is going awry, and although I can't see it affecting anything, I am going to test with those shares removed, just in case.

I'm also going to build Jarvis from source and install it as a portable version (as opposed to the current system-wide install via apt-get) and see if I get the same results running that. There is always the possibility that something untoward occurred when I upgraded from Isengard to Jarvis and that either nothing flagged up or I didn't spot it at the time. I realise that this may be a little straw-clutching but I guess it depends on what the Jarvis build does.

Off to github then !! - I'll post the results later

[DaveBlake]

A ha, scraping just gave me the Bee Gees biog and artwork for Afro Celt Sound System!
I can get an issue, it is not just you, good, I can dig better now

[EDIT] Oh poop, it found the NFO that I had from you originally, hence the Bee Gees.
In a odd place, but it still found it.

What did your system scan for Adam reveal?

[Edit2]
So I started from scratch again, renamed that NFO first, scanned album into empty lib and then scraped artists. No corruptions although the scraper didn't find all the artists Intersting thing was the debug log said

DEBUG: Found matching nfo file: C:\teststuff\black_eagle\artist.nfo
ERROR: Unable to find an url in nfo file: C:\teststuff\black_eagle\artist.nfo

The "found" must be using cache because that file does not exist anymore. Yet a few lines above that which produces the "Found matching nfo file" the cache is cleared. And I can't get it to do it again.

Grrrr. Not getting anywhere with this, hope you fare better

[black_eagle]

Hummm, well I am and I aren't !!!

I built Jarvis v16.1 RC from source and installed it into a directory, ran it in portable mode with the exact same settings as the system-wide version and the same filepath for the test albums. Everything scanned in correctly. Admittedly this was using sqlite rather than SQL but I honestly can't see that making a difference. I will however test that as well just to eliminate it.

So, the issue is with the system-wide install. As the system directory it's installed into doesn't normally have write permissions and Kodi never runs with elevated permissions I can't see there being anything in there that could cause it. Besides, I have checked the appropriate directories and they appear absolutely fine.

Therefore, this leaves just the userdata directory as a possible source. I'm going to move the current userdata somewhere else and let Kodi re-create it fresh then scan again and see if the issue persists.

It has occurred to me that I could use my portable build to scrape the entire Music directory into an SQL db and then point the system-wide install at that. This would remove our friend Adam, but it wouldn't tell us anything. And I want to know what's happening. The other thing I have considered is installing the RC build over the top of 16.0 . This may or may not fix the issue, depending obviously on where the issue actually is. It leads me to a question though. (It's been a long time since I did any serious programming, and even then it was straight C not C++ so bear with me here!)

In "MusicInfoScanner.cpp" at line 1297 there is

PHP Code:

CLog::Log(LOGDEBUG,"Found matching nfo file: %s", CURL::GetRedacted(strNfo).c_str()); 


this just currently returns 'artist.nfo'. If it could be altered to return the full path of the .nfo file its reading, then I would know immediately where Kodi had pulled the information from as it would be in the log. I don't know enough to do it on my own, but I'm guessing that 'GetRedacted' is chopping off the rest of the path ?? I have no idea though how to re-write it. Possibly "CURL:trNfo.c_str()" ? Dunno.

See, I did say this isn't a language I'm at all used to and it's only from my (small) knowledge of other languages that I can even make a stab at guessing what may help.

I ask because a) I'm concerned that if I install 16.1RC system-wide that the problem may still persist and b) because even if it turns out that the issue is with the userdata directory, a build that can identify exactly where it's pulling data from will not only reveal the exact issue, but may also help others in the future if, god forbid, they ever encounter anything like this !!!

Right, off now to move userdata somewhere else. This will have to be a quick test though as there is quite a bit of stuff shared from there and the wife is suffering from "loss of music library" already !!


***EDIT***

Removed the entire .Kodi directory, which is like starting from fresh. Same settings as portable version, same file-path. Adam appears !!!! Go figure !!

[black_eagle]

HAHAHA !!!!!

I have found it!!!! Bloody well found it !!

This may be only a Linux issue, and it may only be with 16.0 (although actually, running in portable mode changes quite a few paths, so it may not be).

Anyway, it is a path issue. But, it's nothing to do with the actual library path. I have no idea if this is reproducible under windows, but on a Linux machine, each user has a 'home' directory. On this machine it's '/home/xbmc/'. To any user other than root, their own home directory appears as just 'Home'. Kodi's userdata etc is in a directory named '.kodi' in the Home directory (the .just means its hidden normally). So, path to userdata is "Home/.kodi/userdata" etc.etc. Now, for some reason, the MusicScanner is reading the top level directory, I.E "Home" and finding an artist.nfo file in it and matching that to multiple artists. I don't know why this is happening but it's totally reproducible on my machine.

Remove that 'artist.nfo' from my home directory and everything scans fine, re-create it and the artist info it contains appears in multiple locations in the DB.

I can't tell you why Kodi is reading the artist.nfo from the home directory, only that it definitely is. I have repeated the test multiple times with different artist.nfo files and each time, as soon as the scanner hits Sinead O'Connor, it matches the nfo file in the home directory to her.

The path that the scanner should start from in my case is "NFS://192.168.1.50/media/sde1/Music/Capture 1995-2010" . How it obtains a local, not a network path is beyond me because there are no exports on the home directory. I do also use SAMBA to share directories to my kids and the userdata content is also over smb, but the root 'Home' directory isn't shared anywhere.

In case you need any more path information (hope I'm not being condescending here) the Kodi binary is in /usr/lib/kodi/ and was installed via apt-get from team-kodi's PPA.


Actually, this is also reproducible when running Kodi in portable mode. I installed it to 'jarvis'. Dropping an 'artist.nfo' into that directory reproduces the error. Path in this case to the executable is '/jarvis/lib/kodi' and is started by calling './jarvis/bin/kodi'.

In both cases it appears that jarvis is defaulting to what it considers as the 'root' directory when looking for an artist.nfo file.

[DaveBlake]

I have found it!!!! Bloody well found it !!

Well done Black_eagle

Glad that you persisted. I was kind of on track when I said that the NFO file with Adam Ant could be somewhere unexpected. The artist scraper trys to find a common path for an artist, and looks there for NFO. Quite how that turned into home I'm not so sure, but I will look at it now I know where to focus.

There was a bit of a clue to the location of artist.nfo in that it doesn't show a path for you in the debug log. In my test that found the Bee Gees info, the debug did list the full path. So not sure we need a code change here, just to understand that no path means home.

Also thanks for the Linux install path information, not condescending at all. I dev under Windows, and run my domestic system on a RPi2 with music files on a NAS drive. I am not that familar with Kodi on Linux, so all info is welcome. Back in the day I have done work on Unix, so useful bits come to me now and then. I will let you know if I can repeat this on Windows.

I have not actually managed to repeat this on Windows, but I do see what is happening. For any artists that are not album artists, e.g. only featured on songs but not given album credit, CmusicDatabase.GetArtistPath returns false and path is left empty. The subsequent call to XFILE::CFile::Exists with just "artist.nfo" must look (on Linux anyway) at home. If there is an artist.nfo file there then its contents will be applied to every artist without an album. I'm sure that is inadvertant behaviour.

This whole escapade convinces me that artist scraping and NFO handling needs some rework. The problems:
1) It can locate an NFO in an unexpected place and cause havoc.

2) It can not handle artist NFO for albums with multiple artists e.g. composer, orchestra, conductor, soloist/choir

3) It can not handle artist NFO for song artists that do not have an album (thus folder)

4) It scrapes info for all song artists (associated with a path/media source) even if you have albumartistonly enabled and never see the song artists or the info that you have needlessly scraped

I question that common path alone is a useful way to determine what artists get scraped and what scraper settings to apply. It works for video - TV, Films etc. are likely to be on different paths. It even makes some sense for albums using different folder/paths for different genres say. But with artists the folder hierarchy idea can easily break, not all artists have folders. OK every artist in the library is related to a number of music files, and they all have a path, hence a common path can be determined. But that common path could be home!

I guess I would like to be able to select the artists that get scraped, but I doubt the UI addition required would be acceptable. I would also like to combine artist.nfo, like import, rather than have a separate file per artist/folder or at least name it with the artist name so can have more than one per folder, or even put them all in one place.

I guess I should start a new thread on scraper improvements

[black_eagle]

I'd just like to say a massive Thank You !! to you Dave. If it hadn't been for you sticking with me through this, and the fact that Kodi is by far the best media centre software available, I could easily have binned it.

If I'd been a new user, I think I'd have definitely been put off.

Good to know that the full path is returned in the debug log. Perhaps I should have spotted it earlier then ? I'm still not sure how that NFO got in there. Possibly I was editing it and saved it in there thinking it was out of the path and therefore out of the way! Curiously, Krypton didn't pick it up. Perhaps some file/path handling routines have been changed.

This whole escapade convinces me that artist scraping and NFO handling needs some rework. The problems:
1) It can locate an NFO in an unexpected place and cause havoc.

2) It can not handle artist NFO for albums with multiple artists e.g. composer, orchestra, conductor, soloist/choir

3) It can not handle artist NFO for song artists that do not have an album (thus folder)

4) It scrapes info for all song artists (associated with a path/media source) even if you have albumartistonly enabled and never see the song artists or the info that you have needlessly scraped

Yeah, 4 annoys me immensely. My wife owns every Now! album since they started. Several of these are Vinyl and are carefully packed away but most of them are CD's stashed in our sideboard. She only plays them now through Kodi but it takes an age to scrape all 92 of them. And then there's the decades one, #1's etc etc.

3 I wasn't aware of. If you turn off albumartistsonly, I believe it creates folders for all the artists does it not, with associated artwork etc.? Certainly it used to, as I found out when I scraped once with it turned off and was horrified by what it did to my neatly organised directory system !!

2 Seems better to me than older versions, or perhaps my improved tagging has helped.

1 Yeah, so it seems

Once again, a big thanks to you - I've added a [solved] to the thread title for anyone else who may encounter this. I'll leave you in peace now. For a while anyway

[DaveBlake]

Helping you has been a pleasure, let me know if you have any other music issues. I think we can both be forgiven for not knowing that an artist.nfo file in home would be applied to every artist without an album!!!

The scraper/NFO problems:

4) Not only slows things for users, but also places unncessary demands on the music data providers which leads to more 503 errors for data that is wanted.

3) Scraping online data for song only artists fine, but artist.nfo relies on an artist>album>song folder structure. All those artists on Now XX do not have a folder each for individual artist.nfo files. I think you are remembering a surprize experience with export.

2) The improvements you see with multiple artists are all down to your improved tagging. It is like 3), say I have a folder for Beethoven with all my albums of his work beneath, any artist.nfo will be for him, so where can I put the info for the conductor or orchestra? Scraping info online is OK, but local data not possible.

Work to be done, if I can get a consensus on how we want scraping to work. Bug chasing is so much easier than getting agreement for improvements.

[scott967]

While on the subject of artist.nfo handling, I find when I do a library export to multiple files I end up with an artist.nfo in the topmost source folder. I'm only interested in the album.nfo files and just delete any and all artist.nfo files so I never investigated where this artist.nfo file comes from.

scott s.
.

[helta]

Yup that's my experience too, that's why I just refuse to export to separate files anymore. I've found its incredibly broken especially when using with mbid's in your tags.