local db scraper - Printable Version

local db scraper - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Development (https://forum.kodi.tv/forumdisplay.php?fid=32)
+--- Forum: Scrapers (https://forum.kodi.tv/forumdisplay.php?fid=60)
+--- Thread: local db scraper (/showthread.php?tid=265342)

local db scraper - keitolainen - 2016-03-21

Hello,

I'm trying to create a scraper to read from a local db, and I'm having some issues.

Here's a bit of background first... I want to use anidb to look up anime shows and movies, but I immediately get banned when I use the provided anidb scraper. So instead I wrote a program to look up show data for given files and store it locally, then wrote another program which serves this data in a kodi-friendly way. The final step is writing a kodi scraper to read from this local db.

However, I'm having absolutely no luck getting this to work.

For instance, if I have a file called "ninja scroll (bluray - 1080p).mkv", I would look up the data using "http://linux-media:8080/animelookup/anime?filename=ninja scroll (bluray - 1080p)", where linux-media is a computer on my lan specified in /etc/hosts. This returns:

Code:
<?xml version="1.0"?><details><title>Juubee Ninpuuchou</title><year>1993</year><rating>6.2</rating><thumb>http://img7.anidb.net/pics/anime/42966.jpg</thumb><poster>http://img7.anidb.net/pics/anime/42966.jpg</poster><plot>Feudal Japan ? a time of danger, intrigue and deception. Kibagami Juubee is a masterless ninja who travels the land alone, hiring his services to those with gold... or a worthy cause. His fearsome abilities have served him well, but a hideous plot to overthrow the government threatens to end his wandering ways and possibly his life. When a small village succumbs to a terrible plague, a team of ninjas are sent to investigate, and realize that all is not as it seems. Ambushed, they are wiped out by a terrifying man-monster with incredible powers, leaving only one alive ? Kagero, a beautiful female ninja, whose touch can bring instant death. Juubee saves her from a fate worse than death, and unwittingly becomes drawn into the web of treachery. He is soon faced with his greatest challenge, an enemy for whom death holds no fear...</plot><genre>violence</genre><genre>samurai</genre><genre>past</genre><genre>ninja</genre><genre>martial arts</genre><genre>action</genre><genre>Japan</genre><genre>Asia</genre><genre>nudity</genre><genre>horror</genre></details>

My scraper looks like this:

Code:
<?xml version="1.0" encoding="UTF-8"?>

<scraper framework="1.1" date="2013-04-04">

    <NfoUrl dest="3">

        <RegExp input="$$1" output="\1" dest="3">

            <expression />

        </RegExp>

    </NfoUrl>

    <CreateSearchUrl dest="3">

        <RegExp input="$$1" output="&lt;url&gt;http://linux-media:8080/animelookup/anime?search=\1&lt;/url&gt;" dest="3">

            <expression/>

        </RegExp>

    </CreateSearchUrl>

    <GetSearchResults dest="8">

        <RegExp input="$$1" output="&lt;?xml version=&quot;1.0&quot; encoding=&quot;iso-8859-1&quot; standalone=&quot;yes&quot;?&gt;&lt;results&gt;&lt;entity&gt;&lt;title&gt;\1&lt;/title&gt;&lt;url&gt;http://linux-media:8080/animelookup/anime?filename=\1&lt;/url&gt;&lt;/entity&gt;&lt;/results&gt;" dest="8">

            <expression>.*&lt;body&gt;(.*)&lt;/body&gt;.*</expression>

        </RegExp>

    </GetSearchResults>

    <GetDetails dest="3">

        <RegExp input="$$1" output="\1" dest="3">

            <expression>.*&lt;body&gt;(.*)&lt;/body&gt;.*</expression>

        </RegExp>

    </GetDetails>

</scraper>

I've edited advancedsettings.xml to add this line to the tvshowmatching tag (prepend):

Code:
<regexp>([^\/]+)$</regexp>

This isn't working and I'm not sure why. Kodi's logfile says there was an error processing the xml. I'm sure it's something really noobish. I can change the output of my server if that's complicating things.

Does anyone have any suggestions for me? Any help would be appreciated. Thanks.

RE: local db scraper - keitolainen - 2016-03-22

I got it. Ends up kodi doesn't like being passed xml, so I changed the server output and fixed the scraper to process it like a regular site. Works perfectly now.