How to get a webpage with Lua Socket ?

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

How to get a webpage with Lua Socket ?

Geoff Smith

Hi


I have used Lua ocket to get a few web pages downloaded in the past without a problem, but have hit a snag with a specific webpage I am trying to download.

I am trying to extract some data from the Betfair homepage.


local webText = http.request([[https://www.betfair.com/exchange/]])


I get back a small chunk of html which contains the text "The document you requested has moved to a new location.  The new location is "https://www.betfair.com/exchange/"


This is the exact url link I passed to the function, so cant understand why the problem is occurring.  If I view that site in any browser it displays the full web page without needing me to register or login or anything like that.


So the Server is detecting I am not using a browser and not giving me the page I require. I have tried tinkering with the headers sent by LuaSocket to make it look more like a Firefox request but to no avail.


Could anyone offer a suggestion as to what I need to do to fix this problem please.


Thanks for any assistance.


Regards Geoff

Reply | Threaded
Open this post in threaded view
|

Re: How to get a webpage with Lua Socket ?

Choonster TheMage
On 25 November 2015 at 06:47, Geoff Smith <[hidden email]> wrote:

> Hi
>
>
> I have used Lua ocket to get a few web pages downloaded in the past without
> a problem, but have hit a snag with a specific webpage I am trying to
> download.
>
> I am trying to extract some data from the Betfair homepage.
>
>
> local webText = http.request([[https://www.betfair.com/exchange/]])
>
>
> I get back a small chunk of html which contains the text "The document you
> requested has moved to a new location.  The new location is
> "https://www.betfair.com/exchange/"
>
>
> This is the exact url link I passed to the function, so cant understand why
> the problem is occurring.  If I view that site in any browser it displays
> the full web page without needing me to register or login or anything like
> that.
>
>
> So the Server is detecting I am not using a browser and not giving me the
> page I require. I have tried tinkering with the headers sent by LuaSocket to
> make it look more like a Firefox request but to no avail.
>
>
> Could anyone offer a suggestion as to what I need to do to fix this problem
> please.
>
>
> Thanks for any assistance.
>
>
> Regards Geoff

Are you sure the page still exists at that URL? I tried opening it
with Chrome and Internet Explorer and downloading with LuaSocket and
LuaSec and got the same "Sorry, we can't find this page" message each
time.

Regards,
Choonster

Reply | Threaded
Open this post in threaded view
|

Re: How to get a webpage with Lua Socket ?

Paul K-2
In reply to this post by Geoff Smith
Hi Geoff,

> local webText = http.request([[https://www.betfair.com/exchange/]])

You should be using https.request (note *https*) after you do "local
https = require("ssl.https")". http.request actually connects to port
80 (so it does http:, even though it looks to you like https:) and you
get redirected to https:, which looks like the same URL.

Paul.

On Tue, Nov 24, 2015 at 11:47 AM, Geoff Smith <[hidden email]> wrote:

> Hi
>
>
> I have used Lua ocket to get a few web pages downloaded in the past without
> a problem, but have hit a snag with a specific webpage I am trying to
> download.
>
> I am trying to extract some data from the Betfair homepage.
>
>
> local webText = http.request([[https://www.betfair.com/exchange/]])
>
>
> I get back a small chunk of html which contains the text "The document you
> requested has moved to a new location.  The new location is
> "https://www.betfair.com/exchange/"
>
>
> This is the exact url link I passed to the function, so cant understand why
> the problem is occurring.  If I view that site in any browser it displays
> the full web page without needing me to register or login or anything like
> that.
>
>
> So the Server is detecting I am not using a browser and not giving me the
> page I require. I have tried tinkering with the headers sent by LuaSocket to
> make it look more like a Firefox request but to no avail.
>
>
> Could anyone offer a suggestion as to what I need to do to fix this problem
> please.
>
>
> Thanks for any assistance.
>
>
> Regards Geoff

Reply | Threaded
Open this post in threaded view
|

Re: How to get a webpage with Lua Socket ?

Geoff Smith
In reply to this post by Choonster TheMage
Hi

Thanks for the reply. Betfair is a multi million pound gambling and trading site so yes it definitely exists and is live right now.

However as it is gambling related, it is not legal in all countries. See http://www.betminded.com/countries-betfair-legal-or-banned-8091.html

I am guessing you are maybe in one of the countries where it is not allowed ?

I wonder if that is maybe related to why I get the puzzling message ? The server needs to know from what country the request is originating ?

Just noticed Paul K has replied too, and it looks like I am not using the correct call, I will go and try Pauls suggestions now

Regards Geoff

________________________________________
From: [hidden email] <[hidden email]> on behalf of Choonster TheMage <[hidden email]>
Sent: 24 November 2015 19:57
To: Lua mailing list
Subject: Re: How to get a webpage with Lua Socket ?

On 25 November 2015 at 06:47, Geoff Smith <[hidden email]> wrote:

> Hi
>
>
> I have used Lua ocket to get a few web pages downloaded in the past without
> a problem, but have hit a snag with a specific webpage I am trying to
> download.
>
> I am trying to extract some data from the Betfair homepage.
>
>
> local webText = http.request([[https://www.betfair.com/exchange/]])
>
>
> I get back a small chunk of html which contains the text "The document you
> requested has moved to a new location.  The new location is
> "https://www.betfair.com/exchange/"
>
>
> This is the exact url link I passed to the function, so cant understand why
> the problem is occurring.  If I view that site in any browser it displays
> the full web page without needing me to register or login or anything like
> that.
>
>
> So the Server is detecting I am not using a browser and not giving me the
> page I require. I have tried tinkering with the headers sent by LuaSocket to
> make it look more like a Firefox request but to no avail.
>
>
> Could anyone offer a suggestion as to what I need to do to fix this problem
> please.
>
>
> Thanks for any assistance.
>
>
> Regards Geoff

Are you sure the page still exists at that URL? I tried opening it
with Chrome and Internet Explorer and downloading with LuaSocket and
LuaSec and got the same "Sorry, we can't find this page" message each
time.

Regards,
Choonster


Reply | Threaded
Open this post in threaded view
|

Re: How to get a webpage with Lua Socket ?

Geoff Smith
In reply to this post by Paul K-2
Hello Paul

Thanks very much for the reply.

I get  "module 'ssl.https' not found:" When I try

https = require("ssl.https")".

I should have mentioned I am using LuaSocket 2.0.2.  Do I need to get a more up to date version of LuaSocket, or is 2.0.2 OK and I need to download an additional SSL library ?

Regards Geoff

________________________________________
From: [hidden email] <[hidden email]> on behalf of Paul K <[hidden email]>
Sent: 24 November 2015 20:03
To: Lua mailing list
Subject: Re: How to get a webpage with Lua Socket ?

Hi Geoff,

> local webText = http.request([[https://www.betfair.com/exchange/]])

You should be using https.request (note *https*) after you do "local
https = require("ssl.https")". http.request actually connects to port
80 (so it does http:, even though it looks to you like https:) and you
get redirected to https:, which looks like the same URL.

Paul.

On Tue, Nov 24, 2015 at 11:47 AM, Geoff Smith <[hidden email]> wrote:

> Hi
>
>
> I have used Lua ocket to get a few web pages downloaded in the past without
> a problem, but have hit a snag with a specific webpage I am trying to
> download.
>
> I am trying to extract some data from the Betfair homepage.
>
>
> local webText = http.request([[https://www.betfair.com/exchange/]])
>
>
> I get back a small chunk of html which contains the text "The document you
> requested has moved to a new location.  The new location is
> "https://www.betfair.com/exchange/"
>
>
> This is the exact url link I passed to the function, so cant understand why
> the problem is occurring.  If I view that site in any browser it displays
> the full web page without needing me to register or login or anything like
> that.
>
>
> So the Server is detecting I am not using a browser and not giving me the
> page I require. I have tried tinkering with the headers sent by LuaSocket to
> make it look more like a Firefox request but to no avail.
>
>
> Could anyone offer a suggestion as to what I need to do to fix this problem
> please.
>
>
> Thanks for any assistance.
>
>
> Regards Geoff


Reply | Threaded
Open this post in threaded view
|

Re: How to get a webpage with Lua Socket ?

Choonster TheMage
In reply to this post by Geoff Smith
On 25 November 2015 at 07:14, Geoff Smith <[hidden email]> wrote:

> Hi
>
> Thanks for the reply. Betfair is a multi million pound gambling and trading site so yes it definitely exists and is live right now.
>
> However as it is gambling related, it is not legal in all countries. See http://www.betminded.com/countries-betfair-legal-or-banned-8091.html
>
> I am guessing you are maybe in one of the countries where it is not allowed ?
>
> I wonder if that is maybe related to why I get the puzzling message ? The server needs to know from what country the request is originating ?
>
> Just noticed Paul K has replied too, and it looks like I am not using the correct call, I will go and try Pauls suggestions now
>
> Regards Geoff
>

I'm in Australia, it's licensed here. I can view the rest of the site,
but not that page.

Regards,
Choonster

Reply | Threaded
Open this post in threaded view
|

Re: How to get a webpage with Lua Socket ?

Tim Channon
In reply to this post by Geoff Smith
I think this is very simple.

You are doing an HTTP request.

The server responds with (in effect, sling your hook, use HTTPS), and
redirects to to HTTPS.

LuaSocket does not as standard do HTTPS, requires an add on. You have to
use HTTPS by some means.

This is perfectly normal web behaviour.

On 24/11/2015 19:47, Geoff Smith wrote:

> Hi
>
>
> I have used Lua ocket to get a few web pages downloaded in the past
> without a problem, but have hit a snag with a specific webpage I am
> trying to download.
>
> I am trying to extract some data from the Betfair homepage.
>
>
> local webText = http.request([[https://www.betfair.com/exchange/]])
>
>
> I get back a small chunk of html which contains the text "The
> document you requested has moved to a new location.  The new location
> is "https://www.betfair.com/exchange/"
>
>
> This is the exact url link I passed to the function, so cant
> understand why the problem is occurring.  If I view that site in any
> browser it displays the full web page without needing me to register
> or login or anything like that.
>
>
> So the Server is detecting I am not using a browser and not giving me
> the page I require. I have tried tinkering with the headers sent by
> LuaSocket to make it look more like a Firefox request but to no
> avail.
>
>
> Could anyone offer a suggestion as to what I need to do to fix this
> problem please.
>
>
> Thanks for any assistance.
>
>
> Regards Geoff
>


Reply | Threaded
Open this post in threaded view
|

Re: How to get a webpage with Lua Socket ?

Choonster TheMage
In reply to this post by Geoff Smith
On 25 November 2015 at 07:23, Geoff Smith <[hidden email]> wrote:

> Hello Paul
>
> Thanks very much for the reply.
>
> I get  "module 'ssl.https' not found:" When I try
>
> https = require("ssl.https")".
>
> I should have mentioned I am using LuaSocket 2.0.2.  Do I need to get a more up to date version of LuaSocket, or is 2.0.2 OK and I need to download an additional SSL library ?
>
> Regards Geoff
>

ssl.https is provided by LuaSec:
https://github.com/brunoos/luasec

Regards,
Choonster

Reply | Threaded
Open this post in threaded view
|

Re: How to get a webpage with Lua Socket ?

Geoff Smith
In reply to this post by Choonster TheMage
Hi

Ah Interesting, yes betfair is legal in Aus, but I know there are some very specific conditions imposed by the Australian government. They stipulate the transactions have to occur on physical Servers based in Aus (Tasmania if I recall).  So there is a legal Iron curtain that keeps Betfair Aus separate from the main Betfair site. I bet that is the reason you cant see the standard Betfair site.

Thanks for the luasec linky

Geoff

________________________________________
From: [hidden email] <[hidden email]> on behalf of Choonster TheMage <[hidden email]>
Sent: 24 November 2015 20:24
To: Lua mailing list
Subject: Re: How to get a webpage with Lua Socket ?

On 25 November 2015 at 07:14, Geoff Smith <[hidden email]> wrote:

> Hi
>
> Thanks for the reply. Betfair is a multi million pound gambling and trading site so yes it definitely exists and is live right now.
>
> However as it is gambling related, it is not legal in all countries. See http://www.betminded.com/countries-betfair-legal-or-banned-8091.html
>
> I am guessing you are maybe in one of the countries where it is not allowed ?
>
> I wonder if that is maybe related to why I get the puzzling message ? The server needs to know from what country the request is originating ?
>
> Just noticed Paul K has replied too, and it looks like I am not using the correct call, I will go and try Pauls suggestions now
>
> Regards Geoff
>

I'm in Australia, it's licensed here. I can view the rest of the site,
but not that page.

Regards,
Choonster


Reply | Threaded
Open this post in threaded view
|

Re: How to get a webpage with Lua Socket ?

Jonathan Goble
On Tue, Nov 24, 2015 at 3:36 PM, Geoff Smith <[hidden email]> wrote:
> Ah Interesting, yes betfair is legal in Aus, but I know there are some very specific conditions imposed by the Australian government. They stipulate the transactions have to occur on physical Servers based in Aus (Tasmania if I recall).  So there is a legal Iron curtain that keeps Betfair Aus separate from the main Betfair site. I bet that is the reason you cant see the standard Betfair site.

I can see the Betfair website just fine here in the U.S., where online
gambling is most definitely illegal. I just can't register an account.

Reply | Threaded
Open this post in threaded view
|

Re: How to get a webpage with Lua Socket ?

Geoff Smith
In reply to this post by Paul K-2
Hi Paul

I downloaded luasec, and hit the usual type of problem I get with Lua.  (I like Lua, but this is my pet peeve with it !)

This library doesnt appear to have a windows binary available.  So I have to build it myself, with not very good build notes.

I then discover it uses openssl C library, so I now have to figure out where to download openssl and which version I need to get to be compatible, and where all the files need to go to build it etc etc.

What sounded like a quick Lua script is already not straightforward and probably more hassle than just writing this in C# in the first place.  ( I could probably find a well documented online example of how to do this for C#)

I wonder if I could  snag the luasec  *.lua files and *.dll file I need from my  Zerobrane Studio installation ?
That might be a simpler solution ?

Geoff

________________________________________
From: [hidden email] <[hidden email]> on behalf of Paul K <[hidden email]>
Sent: 24 November 2015 20:03
To: Lua mailing list
Subject: Re: How to get a webpage with Lua Socket ?

Hi Geoff,

> local webText = http.request([[https://www.betfair.com/exchange/]])

You should be using https.request (note *https*) after you do "local
https = require("ssl.https")". http.request actually connects to port
80 (so it does http:, even though it looks to you like https:) and you
get redirected to https:, which looks like the same URL.

Paul.

On Tue, Nov 24, 2015 at 11:47 AM, Geoff Smith <[hidden email]> wrote:

> Hi
>
>
> I have used Lua ocket to get a few web pages downloaded in the past without
> a problem, but have hit a snag with a specific webpage I am trying to
> download.
>
> I am trying to extract some data from the Betfair homepage.
>
>
> local webText = http.request([[https://www.betfair.com/exchange/]])
>
>
> I get back a small chunk of html which contains the text "The document you
> requested has moved to a new location.  The new location is
> "https://www.betfair.com/exchange/"
>
>
> This is the exact url link I passed to the function, so cant understand why
> the problem is occurring.  If I view that site in any browser it displays
> the full web page without needing me to register or login or anything like
> that.
>
>
> So the Server is detecting I am not using a browser and not giving me the
> page I require. I have tried tinkering with the headers sent by LuaSocket to
> make it look more like a Firefox request but to no avail.
>
>
> Could anyone offer a suggestion as to what I need to do to fix this problem
> please.
>
>
> Thanks for any assistance.
>
>
> Regards Geoff


Reply | Threaded
Open this post in threaded view
|

Re: How to get a webpage with Lua Socket ?

Paul K-2
Hi Geoff,

> This library doesnt appear to have a windows binary available.  So I have to build it myself, with not very good build notes.
> I wonder if I could  snag the luasec  *.lua files and *.dll file I need from my  Zerobrane Studio installation ? That might be a simpler solution ?

Yes, you should be able to get all the files including the binary from
this commit: https://github.com/pkulchenko/ZeroBraneStudio/commit/0d8e6b058186fb361680ed05ff8dceec5123c70c.
The DLL is no longer packaged with the IDE, but you can still get it
from the archive. Note that you still need openssl libraries installed
and referenced in PATH, but they come with many applications, like
wget, svn, and others.

Another option is to compile luasec using the build scripts from ZBS.
You'll need to pull the latest changes from the repository and then
run "(cd build; bash build-win32.sh lua luasec)". This should build
and install ssl.dll in the right place under ZBS tree (and it will
include all the dependencies) as long as you have mingw/msys or tdm
installed.

Yet another option is to try luarocks or luadist
(https://github.com/LuaDist/luasec/releases/tag/0.4-Windows-x86);
you'd still need openssl libraries and headers though.

Paul.

Reply | Threaded
Open this post in threaded view
|

Re: How to get a webpage with Lua Socket ?

slayer beast
luasocket is old and unmaintained. it shouldn't be used for new projects.
i've been using lua-curl since a while and it works great for me. it
supports ssl out of the box and can do async too.


On Tue, Nov 24, 2015 at 7:03 PM, Paul K <[hidden email]> wrote:

> Hi Geoff,
>
>> This library doesnt appear to have a windows binary available.  So I have to build it myself, with not very good build notes.
>> I wonder if I could  snag the luasec  *.lua files and *.dll file I need from my  Zerobrane Studio installation ? That might be a simpler solution ?
>
> Yes, you should be able to get all the files including the binary from
> this commit: https://github.com/pkulchenko/ZeroBraneStudio/commit/0d8e6b058186fb361680ed05ff8dceec5123c70c.
> The DLL is no longer packaged with the IDE, but you can still get it
> from the archive. Note that you still need openssl libraries installed
> and referenced in PATH, but they come with many applications, like
> wget, svn, and others.
>
> Another option is to compile luasec using the build scripts from ZBS.
> You'll need to pull the latest changes from the repository and then
> run "(cd build; bash build-win32.sh lua luasec)". This should build
> and install ssl.dll in the right place under ZBS tree (and it will
> include all the dependencies) as long as you have mingw/msys or tdm
> installed.
>
> Yet another option is to try luarocks or luadist
> (https://github.com/LuaDist/luasec/releases/tag/0.4-Windows-x86);
> you'd still need openssl libraries and headers though.
>
> Paul.
>

Reply | Threaded
Open this post in threaded view
|

Re: How to get a webpage with Lua Socket ? (Solved)

Geoff Smith
In reply to this post by Paul K-2
Hi

Thanks Paul and everyone that helped with suggestions. After a fair bit of struggle I got it working.  Ad hoc and messy process I know, but at least it worked.

I didnt really like any of those solutions, none were straightforward. For the average Joe Win programmer like me, I dont like the LuaDist type solution of download all the dependency sources and download mingw and compile ourselves.  Too much hassle and too many things likely to trip you up.

For anyone that struggles with this in the future, here is exactly what I did to get luasec/SSL working  (I am using Lua 5.1 and Win7 64 Bit for reference)

I am using LuaSocket 2.0.2 from an earlier LuaForWindows Install
Downloaded Luasec 0.5 from Github

Refer to this page to get the Windows Luasec binaries.          https://love2d.org/forums/viewtopic.php?f=5&t=76728&p=159477&hilit=linux+binary
I  chose Windows x86 (32-bit)

 Copied the 'ssl.lua' file to a folder in my LUA_PATH.
 Copied the 'ssl.dll' file to a folder in  my LUA_CPATH.
 Created a directory 'ssl' in my LUA_PATH and copied 'https.lua' to it.

Tested it with this code
-----------------------------------------------------------------------------------------------------------------------
require("socket")
local https = require("ssl.https")

local body, code, headers, status = https.request("https://en.wikipedia.org/wiki/HTTPS")
print(body, code, headers, status)
-----------------------------------------------------------------------------------------------------------------------

I got a Window dialogue box pop up with a fairly obscure sounding error. Forget the exact wording, something like

"The ordinal 350 could not be located in the dynamic link library SSLEAY32.dll"

That one had me foxed for a while, but found an article.  http://www.techchattr.com/openssl-working-windows-ssleay32-dll-error

As the article points out the error was due to the version of openSSL that ships and installs with WAMP.
(I had installed WAMP on my PC some time ago)

So then went to this page    http://slproweb.com/products/Win32OpenSSL.html

and downloaded and installed   Win32 OpenSSL v1.0.2d

That fixed the problem, I can now download https pages successfully.

My code to get the live football scores from the Betfair page is not working but thats just some html parsing logic problem I daresay.
I should be able to debug that and get it working tomorrow with some assistance from Zerobrane Studio.

Hey Paul, you should try Zerobrane Studio, its excellent and very helpful at  bug finding !  :)

Regards Geoff

________________________________________
From: [hidden email] <[hidden email]> on behalf of Paul K <[hidden email]>
Sent: 24 November 2015 22:03
To: Lua mailing list
Subject: Re: How to get a webpage with Lua Socket ?

Hi Geoff,

> This library doesnt appear to have a windows binary available.  So I have to build it myself, with not very good build notes.
> I wonder if I could  snag the luasec  *.lua files and *.dll file I need from my  Zerobrane Studio installation ? That might be a simpler solution ?

Yes, you should be able to get all the files including the binary from
this commit: https://github.com/pkulchenko/ZeroBraneStudio/commit/0d8e6b058186fb361680ed05ff8dceec5123c70c.
The DLL is no longer packaged with the IDE, but you can still get it
from the archive. Note that you still need openssl libraries installed
and referenced in PATH, but they come with many applications, like
wget, svn, and others.

Another option is to compile luasec using the build scripts from ZBS.
You'll need to pull the latest changes from the repository and then
run "(cd build; bash build-win32.sh lua luasec)". This should build
and install ssl.dll in the right place under ZBS tree (and it will
include all the dependencies) as long as you have mingw/msys or tdm
installed.

Yet another option is to try luarocks or luadist
(https://github.com/LuaDist/luasec/releases/tag/0.4-Windows-x86);
you'd still need openssl libraries and headers though.

Paul.


Reply | Threaded
Open this post in threaded view
|

Re: How to get a webpage with Lua Socket ?

Björn Kalkbrenner
In reply to this post by slayer beast
Hi,

> luasocket is old and unmaintained. it shouldn't be used for new projects.

Uhm? Sure? Just my two cents:

Yes, it's seems it is not full throttle development (are updates needed in it just works?) and i think Diego mentioned somewhere that he hasn't that much time but there was even an Ipv6 capable update/RC and it seems that there are still commits:

https://github.com/diegonehab/luasocket

Mentioning that to the maybe (!) most used lua socket library in the lua eco system (which is also mature, portable and stable)...

I am not sure how lua-curl can be a replacement for luasocket. Maybe for the http.request module - that makes more sence to me :)

Please correct me if i am wrong here.

Bye
Björn


> i've been using lua-curl since a while and it works great for me. it
> supports ssl out of the box and can do async too.
>
>
>> On Tue, Nov 24, 2015 at 7:03 PM, Paul K <[hidden email]> wrote:
>> Hi Geoff,
>>
>>> This library doesnt appear to have a windows binary available.  So I have to build it myself, with not very good build notes.
>>> I wonder if I could  snag the luasec  *.lua files and *.dll file I need from my  Zerobrane Studio installation ? That might be a simpler solution ?
>>
>> Yes, you should be able to get all the files including the binary from
>> this commit: https://github.com/pkulchenko/ZeroBraneStudio/commit/0d8e6b058186fb361680ed05ff8dceec5123c70c.
>> The DLL is no longer packaged with the IDE, but you can still get it
>> from the archive. Note that you still need openssl libraries installed
>> and referenced in PATH, but they come with many applications, like
>> wget, svn, and others.
>>
>> Another option is to compile luasec using the build scripts from ZBS.
>> You'll need to pull the latest changes from the repository and then
>> run "(cd build; bash build-win32.sh lua luasec)". This should build
>> and install ssl.dll in the right place under ZBS tree (and it will
>> include all the dependencies) as long as you have mingw/msys or tdm
>> installed.
>>
>> Yet another option is to try luarocks or luadist
>> (https://github.com/LuaDist/luasec/releases/tag/0.4-Windows-x86);
>> you'd still need openssl libraries and headers though.
>>
>> Paul.
>