Beta - WebGrab+Plus Configurator

Beta WebGrab+Plus Configurator - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Add-on Support (https://forum.kodi.tv/forumdisplay.php?fid=27)
+---- Forum: Program Add-ons (https://forum.kodi.tv/forumdisplay.php?fid=151)
+---- Thread: Beta WebGrab+Plus Configurator (/showthread.php?tid=285918)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

RE: WebGrab+Plus Configurator - donbrew - 2016-08-11

So 2 days every 18 hours, or whatever mathematically works for an overlap. Not much use in getting the same days data 4 times a day.
I had thought that it was 24 hours not 1 calendar day, that makes a difference.

I guess I don't understand channel logos. When I use the xmltv file generated by WebGrab++ on the PC everything from tvguide.co.uk has a logo. When I use the xmltv generated using the configurator there are no logos. I am not really needing logos, but they add some color.
I moved the xmltv folder using the configurator, is that it?

RE: WebGrab+Plus Configurator - primaeval - 2016-08-11

(2016-08-11, 02:26)donbrew Wrote: So 2 days every 18 hours, or whatever mathematically works for an overlap. Not much use in getting the same days data 4 times a day.
I had thought that it was 24 hours not 1 calendar day, that makes a difference.

I guess I don't understand channel logos. When I use the xmltv file generated by WebGrab++ on the PC everything from tvguide.co.uk has a logo. When I use the xmltv generated using the configurator there are no logos. I am not really needing logos, but they add some color.
I moved the xmltv folder using the configurator, is that it?

It looks like someone has messed up the tvguide.co.uk ini file.

On my server I have a version that adds the logos.
It contains the revision line:
* @Revision 14 - [23/02/2016] koori

The one from the current site ini pack has deleted the lines for processing the logos:
* @Revision 14 - [28/07/2016] Blackbear199

The koori one still works.
I have no idea what Blackbear199 was trying to do.

The part with the logo in koori's version is:

Code:
index_showsplit.modify {cleanup(removeduplicates=equal span=1)}                * remove any duplicate show

end_scope

*** Channellogo

index_urlchannellogo.modify {set|http://my.tvguide.co.uk/channel_logos/'config_site_id'.png}

index_start.scrub {regex||<span class=\"season\">(\d{1,2}\:\d{2}(?:am\|pm))||}

RE: WebGrab+Plus Configurator - DarrenHill - 2016-08-11

OK just to update, had a play with the newer versions last night (of both this and the fullscreen TV guide). Now have it fully working and nicely talking to one another, complete with all logo's. It's pulling info from a range of sites (tvguideuk.telegraph.co.uk, radiotimes.com and m.tvguide.co.uk-full) and at the moment it all is working fine (even ITV now I've got around to updating/fixing that add-on).

I've nothing much more to add than Mike's comments yesterday about the possible flags to add, as they cover all the ones that are available (at least all the ones that are detailed in the example file that comes with WebGrab+). Personally I think they are all "nice to have" rather than really required (all the required ones are already in place), although being able to use the proxy setting may help some people.

One thing I would change though is the default timespan to be "1", ie for today and tomorrow. Defaulting to only today ("0") could limit usefulness especially if you're towards the end of the day and/or working across timezones. That said I can understand the desire to minimise both Webgrab runtimes and also load on the servers.

But aside from that, all the add-ons seem to be playing nicely together and working well. Top job Smile

RE: WebGrab+Plus Configurator - primaeval - 2016-08-11

0.0.8
- some more config settings (proxy, postprocess, user-agent, update)
- site ini version
- output log viewer
- default timespan is 2 days now (timespan=1)

The log viewer just uses a select dialog.
If anyone knows how to show a big text box for the log file in Kodi, please let me know.

RE: WebGrab+Plus Configurator - primaeval - 2016-08-11

If anyone wants to mirror the tvguide.co.uk logos they can try this:
"?*/:" are converted to "_".

Code:
import requests

import re

r = requests.get('http://www.tvguide.co.uk/')

html = r.text

match = re.search('<div id="div-add-channel">.*?</select>',html,flags=(re.DOTALL | re.MULTILINE))

match = re.findall(r'<option value=(.*?)>(.*?)</option>',match.group(0))

for (id,name) in match:

    url = "http://my.tvguide.co.uk/channel_logos/%s.png" % id

    new_name = re.sub('[\?\*/:]','_',name)

    png = "%s.png" % new_name

    r = requests.get(url)

    data = r.content

    f = open(png,"wb")

    f.write(data)

    f.close()

    print name

RE: WebGrab+Plus Configurator - MikeKL - 2016-08-11

Updated from 0.0.6 to 0.0.9 and liking the major improvements I see in the addon settings options and other discussed changes. Cool

Currently having issues with writing the config file, "webgrabplus configurator error" message in kodi.

kodi.log

RE: WebGrab+Plus Configurator - primaeval - 2016-08-11

(2016-08-11, 14:05)MikeKL Wrote: Updated from 0.0.6 to 0.0.9 and liking the major improvements I see in the addon settings options and other discussed changes.
Currently having issues with writing the config file, "webgrabplus configurator error" message in kodi.

kodi.log

0.0.10
- <update> fix

The error was from me not setting a default value for the <update> Setting.

There are some other errors in the log that aren't really errors.
I can't remember how to get xmbcswift2 to stop complaining when there is nothing to play or list.
eg

Code:
ERROR: CGUIMediaWindow::GetDirectory(plugin://script.webgrab/write_config) failed

RE: WebGrab+Plus Configurator - donbrew - 2016-08-11

Switched back to koori version got logos now.

RE: WebGrab+Plus Configurator - primaeval - 2016-08-11

0.0.11
- sort secondary criteria (usually by name)

This should stop the need to Move the channels around too much.
If you sort by country now, it will also sort by name.

RE: WebGrab+Plus Configurator - primaeval - 2016-08-11

0.0.12
- Move fix

@MikeKL There was a bug in the Move code for moving channels downwards.
That will teach to put too much trust random pieces of code from the internet.

RE: WebGrab+Plus Configurator - MikeKL - 2016-08-11

Updated to 0.0.12

Config being written fine again Big Grin

I added "automatic" to Proxy and "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; yie9)" to User Agent in editable config settings of add-on and fields are updated correctly within written configuration file. Big Grin

Also looked at adding more channels to my channels list (moving channels in list appeared to be rock solid) also noticed improved Countries listings when drilling down through lists "all in clean alphanumeric order" Cool

Great work primeaval Nod

RE: WebGrab+Plus Configurator - MikeKL - 2016-08-12

My first actual manual run of webgraplus from configurator console Blush

, do we really need for kodi to sit with spinning wheel whilst job is running?

Would be good if we recieved a message stating Job Started then Job Completed or similar.
Then we can carry on doing other activities with Kodi whilst webgrabplus job is manually running.

With the show webgrab+plus log, where does option currently look for log as mine was empty even though log existed in
Userdata\addon_data\service.webgrabplus (possibly need to specify in configurator settings -> paths?)

Like the show site ini version to know when we last downloaded from webgrabplus site Cool

RE: WebGrab+Plus Configurator - primaeval - 2016-08-12

(2016-08-12, 10:23)MikeKL Wrote: My first actual manual run of webgraplus from configurator console , do we really need for kodi to sit with spinning wheel whilst job is running?

Would be good if we recieved a message stating Job Started then Job Completed or similar.
Then we can carry on doing other activities with Kodi whilst webgrabplus job is manually running.

With the show webgrab+plus log, where does option currently look for log as mine was empty even though log existed in
Userdata\addon_data\service.webgrabplus (possibly need to specify in configurator settings -> paths?)

Like the show site ini version to know when we last downloaded from webgrabplus site

I'll try to get the webgrab program to work in the background a bit later.

I think you need to specify this <filename> to match up with the processing command from libreELEC:

Code:
<filename><filename>/storage/.kodi/userdata/addon_data/service.webgrabplus/guide_wgp.xml</filename>

So the folder is

Code:
/storage/.kodi/userdata/addon_data/service.webgrabplus

and the name is

Code:
guide_wgp.xml

in Settings.

The webgrab service uses this command for the time correction:

Code:
mono $ADDON_DIR/WebGrab+Plus/WebGrab+Plus.exe

cp guide_wgp.xml guide.xml

mono $ADDON_DIR/WebGrab+Plus/xmltv_time_correct.exe guide_wgp.xml guide.xml < /dev/zero

mono $ADDON_DIR/WebGrab+Plus/WG2MP.exe guide.xml mediaportal.xml

In storage/.kodi/addons/service.webgrabplus/bin/webgrabplus.start
You can see that it needs the name guide_wgp.xml and creates guide.xml.

There is a bug in the log viewer code. I'll fix it next.
The LibreELEC webgrab program command doesn't take a folder as an argument as the Windows version does, so it uses its default folder.
So, if you want to see the libreelec log (after I fix it), you have to point the Config Output Folder setting to:

Code:
/storage/.kodi/userdata/addon_data/service.webgrabplus

I think this is going to need some automation. It is starting to confuse me. Wink

RE: WebGrab+Plus Configurator - primaeval - 2016-08-12

0.0.13
- background Webgrab program task (notification when finished)
- log location fix

Here is my settings.xml that works with libreelec:
/storage/.kodi/userdata/addon_data/script.webgrab/settings.xml

Code:
<settings>

    <setting id="config_output_folder" value="/storage/.kodi/userdata/addon_data/service.webgrabplus/" />

    <setting id="exe" value="/storage/.kodi/addons/service.webgrabplus/bin/webgrabplus.start" />

    <setting id="mdb" value="false" />

    <setting id="mdb_grab" value="false" />

    <setting id="mdb_run" value="false" />

    <setting id="proxy" value="" />

    <setting id="proxy_password" value="" />

    <setting id="proxy_user" value="" />

    <setting id="rex" value="false" />

    <setting id="rex_grab" value="false" />

    <setting id="rex_run" value="false" />

    <setting id="timespan" value="1" />

    <setting id="update" value="0" />

    <setting id="user_agent" value="" />

    <setting id="xmltv_name" value="guide_wgp.xml" />

    <setting id="xmltv_output_folder" value="/storage/.kodi/userdata/addon_data/service.webgrabplus/" />

</settings>

RE: WebGrab+Plus Configurator - MikeKL - 2016-08-12

0.0.13 Looking very good.. Cool

Background task runnning now, thanks for the info on settings when using the LibreELEC webgrabplus add-on
(Useful check against what I had included, especially the "guide_wgp.xml" setting)

Be aware of need to set "f" (full/force) option in configurator GUI when adding additional channels to get EPG information added to guide for new channels (Reasonably sure they are never added when update set at default)

Got confused around the adding "f" in settings, then remember to write config file, then remember to copy Config to output folder Wink

Lets see if outputs with f setting is to expectations when viewed in TV Guide Fullscreen later this afternoon
(I added a few Radio channels Wink

)

Show WebGrab+Plus log is working Cool

Thanks so much for all these updates to beta WebGrab+Plus Configurator (I can't keep up with you and I am only testing Blush

)