html source retrived...differ by simple wget

html source retrived...differ by simple wget - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Development (https://forum.kodi.tv/forumdisplay.php?fid=32)
+--- Forum: Scrapers (https://forum.kodi.tv/forumdisplay.php?fid=60)
+--- Thread: html source retrived...differ by simple wget (/showthread.php?tid=77366)

html source retrived...differ by simple wget - muttley:bd - 2010-07-17

I hope can explain what make me scraper... Rolleyes

if i get this page http://www.mymovies.it/film/2009/avatar/trailer/ with <url> tag, the html source returned is different by a simple linux wget.

Linux wget: http://paste2.org/p/918238 see line 725
Xbmc scraper: http://paste2.org/p/918257 see line 727

wget:

Code:
so.addVariable('plugins','gapro-1,adtvideo');

so.addVariable('adtvideo.config','http://www.mymovies.it/v9/adv/preroll/config_nestea.asp')

so.addVariable('file','[b]http://pad.mymovies.it/filmclub/2008/03/090/trailer.flv[/b]');

so.addVariable('gapro.accountid', 'UA-259522-1'); 

so.write('mediaspace');

xbmc:

Code:
so.addVariable('plugins','gapro-1');

so.addVariable('file','[b]http://www.mymovies.it/v9/include/player/playlist/editoriale.asp?codicefilm=2008/03/090[/b]');

so.addVariable('repeat','list');

so.addVariable('gapro.accountid', 'UA-259522-1'); 

so.write('mediaspace');

There is other little change, a javacript <script> block from 745 to 764 line, not present in wget version.

I use latest xbmc pre-10.5 r31718

With 'old' xbmc this didn't happen...and also with xbmc4xbox now.
What is browser identify of xbmc?!

any suggestion?!

thanks!

p.s. another question: for submit an addons i must send a mail to: [email protected] whit subject "[Svn pull] xxxx scraper" and in body url to svn for example!?!?

- muttley:bd - 2010-07-20

can i submit as bug?! Sad

...someone for question in ps?!

- jmarshall - 2010-07-21

I suspect your guess is accurate and that the website is returning different things based on useragent. I believe you can spoof the referer at least - not sure about the user-agent.

And yes - ensure you take note of the addons guidelines (eg icon size etc.) then just send an SVN pull request to the mailing list.

Cheers,
Jonathan

- muttley:bd - 2010-07-21

thank you!!!

I'll try with referrer, but it's strange that work good with xbmc 9.11 and xbmc4xbox Eek

Quote:And yes - ensure you take note of the addons guidelines (eg icon size etc.) then just send an SVN pull request to the mailing list.

ok!