Problem trying to download a webpage

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem trying to download a webpage

Geoff Smith
I am trying to download a webpage using Lua 5.1 and Luasec (not sure which version)

I am seeing the same error on several pages, but one example is [[https://www.attheraces.com/racecards]]
I am getting an error stating "sslv3 alert handshake failure"

I am not an expert on the https protocol so not sure what I need to do to fix this problem. I am assuming i need to tell it to use TLS v1.2 or later to make it work. Not proved that this is the problem though. Here is gist of my latest code attempt that still gives me the same error.

ssl.newcontext({mode="client", protocol="tlsv1_2"})

local url = [[https://www.attheraces.com/racecards]]

webText, code, headers, status = https.request({url=url,  protocol="tlsv1_2"})
print(code, headers, status)

I have tried protocol = "sslv23" and "sslv3"  neither fixed the problem

Any help on this issue would be most appreciated, thanks.







Virus-free. www.avast.com
Reply | Threaded
Open this post in threaded view
|

Re: Problem trying to download a webpage

Oliver Kroth


Am 25.03.19 um 12:51 schrieb Geoff Smith:
> ssl.newcontext({mode="client", protocol="tlsv1_2"})
Hi,

you may like to add an options="no_sslv3", which tells your side not to
attempt a SSL3  connection that is not accepted at the server.

--
Oliver

Reply | Threaded
Open this post in threaded view
|

Re: Problem trying to download a webpage

Geoff Smith
Hi

Thanks for the suggestion, I tried

webText, code, headers, status = https.request({url=url, options="no_sslv3", protocol="tlsv1_2"})

This didnt work, I get the same error "sslv3 alert handshake failure"

Do I need a specific version of luasec for that options command ?

Regards Geoff


From: [hidden email] <[hidden email]> on behalf of Oliver Kroth <[hidden email]>
Sent: 25 March 2019 12:19
To: [hidden email]
Subject: Re: Problem trying to download a webpage
 


Am 25.03.19 um 12:51 schrieb Geoff Smith:
> ssl.newcontext({mode="client", protocol="tlsv1_2"})
Hi,

you may like to add an options="no_sslv3", which tells your side not to
attempt a SSL3  connection that is not accepted at the server.

--
Oliver

Reply | Threaded
Open this post in threaded view
|

Re: Problem trying to download a webpage

Dirk Laurie-2
In reply to this post by Geoff Smith
I suppose this does not answer your question, but ... if you don't
explicitly need SSL, you can get away with curl.

I figured out the 'if' below a year or two ago, and now just
cut-and-paste it every time.

Having loaded lua-curl as "curl", this is all:

logins = {}
session = curl.easy()
if curl.OPT_COOKIEFILE and curl.OPT_COOKIEJAR then
    cookies = os.tmpname()
    session:setopt(curl.OPT_COOKIEFILE,cookies)
    session:setopt(curl.OPT_COOKIEJAR,cookies)
    logins[session] = cookies
  else
    message("Your 'curl' does not support cookies. You will be anonymous.")
  end
session:setopt{url="https://www.attheraces.com/racecards",writefunction=print}:perform()

I see the entire HTML code of the landing page.

To store it in a table, just put something like 'function(str)
table.insert(buffer,str) end' instead of 'print' having done
'buffer={}' first.



Op Ma. 25 Mrt. 2019 om 13:51 het Geoff Smith <[hidden email]> geskryf:

>
> I am trying to download a webpage using Lua 5.1 and Luasec (not sure which version)
>
> I am seeing the same error on several pages, but one example is [[https://www.attheraces.com/racecards]]
> I am getting an error stating "sslv3 alert handshake failure"
>
> I am not an expert on the https protocol so not sure what I need to do to fix this problem. I am assuming i need to tell it to use TLS v1.2 or later to make it work. Not proved that this is the problem though. Here is gist of my latest code attempt that still gives me the same error.
>
> ssl.newcontext({mode="client", protocol="tlsv1_2"})
>
> local url = [[https://www.attheraces.com/racecards]]
>
> webText, code, headers, status = https.request({url=url,  protocol="tlsv1_2"})
> print(code, headers, status)
>
> I have tried protocol = "sslv23" and "sslv3"  neither fixed the problem
>
> Any help on this issue would be most appreciated, thanks.
>
>
>
>
>
>
>
> Virus-free. www.avast.com

Reply | Threaded
Open this post in threaded view
|

Re: Problem trying to download a webpage

Geoff Smith
hi

Thanks for the response, I would be happy to use whatever works to download  the page i need. However I have never used lua curl before, so I am unsure  how to get that working.
 
 Could you point me at some Windows library binaries I would need to try this ?  Also what is the require line needed ? require "curl" or "luacurl" "libcurl".  A quick google search seemed to have a mixture of these



From: [hidden email] <[hidden email]> on behalf of Dirk Laurie <[hidden email]>
Sent: 25 March 2019 13:34
To: Lua mailing list
Subject: Re: Problem trying to download a webpage
 
I suppose this does not answer your question, but ... if you don't
explicitly need SSL, you can get away with curl.

I figured out the 'if' below a year or two ago, and now just
cut-and-paste it every time.

Having loaded lua-curl as "curl", this is all:

logins = {}
session = curl.easy()
if curl.OPT_COOKIEFILE and curl.OPT_COOKIEJAR then
    cookies = os.tmpname()
    session:setopt(curl.OPT_COOKIEFILE,cookies)
    session:setopt(curl.OPT_COOKIEJAR,cookies)
    logins[session] = cookies
  else
    message("Your 'curl' does not support cookies. You will be anonymous.")
  end
session:setopt{url="<a href="https://www.attheraces.com/racecards&quot;,writefunction=print}:perform(">https://www.attheraces.com/racecards",writefunction=print}:perform()

I see the entire HTML code of the landing page.

To store it in a table, just put something like 'function(str)
table.insert(buffer,str) end' instead of 'print' having done
'buffer={}' first.



Op Ma. 25 Mrt. 2019 om 13:51 het Geoff Smith <[hidden email]> geskryf:
>
> I am trying to download a webpage using Lua 5.1 and Luasec (not sure which version)
>
> I am seeing the same error on several pages, but one example is [[https://www.attheraces.com/racecards]]
> I am getting an error stating "sslv3 alert handshake failure"
>
> I am not an expert on the https protocol so not sure what I need to do to fix this problem. I am assuming i need to tell it to use TLS v1.2 or later to make it work. Not proved that this is the problem though. Here is gist of my latest code attempt that still gives me the same error.
>
> ssl.newcontext({mode="client", protocol="tlsv1_2"})
>
> local url = [[https://www.attheraces.com/racecards]]
>
> webText, code, headers, status = https.request({url=url,  protocol="tlsv1_2"})
> print(code, headers, status)
>
> I have tried protocol = "sslv23" and "sslv3"  neither fixed the problem
>
> Any help on this issue would be most appreciated, thanks.
>
>
>
>
>
>
>
> Virus-free. www.avast.com


Virus-free. www.avast.com
Reply | Threaded
Open this post in threaded view
|

Re: Problem trying to download a webpage

Dirk Laurie-2
I installed it from LuaRocks, which normally takes care of system
dependencies. It should install on Windows or Unix platforms with just

luarocks install lua-curl

The rockspec says:

external_dependencies = {
  platforms = {
    windows = {
      CURL = {
        header  = "curl/curl.h",
        library = "libcurl",
      }
    };
    unix = {
      CURL = {
        header  = "curl/curl.h",
        -- library = "curl",
      }
    };
  }
}

build = {
  copy_directories = {'doc', 'examples', 'test'},

  type = "builtin",

  platforms = {
    windows = { modules = {
      lcurl = {
        libraries = {"libcurl", "ws2_32"},
      }
    }},
    unix    = { modules = {
      lcurl = {
        libraries = {"curl"},
      }
    }}
  },

Unfortunatly I can't offer Windows-specific advice, but there must be
many list members who can.


Op Ma. 25 Mrt. 2019 om 19:30 het Geoff Smith <[hidden email]> geskryf:

>
> hi
>
> Thanks for the response, I would be happy to use whatever works to download  the page i need. However I have never used lua curl before, so I am unsure  how to get that working.
>
>  Could you point me at some Windows library binaries I would need to try this ?  Also what is the require line needed ? require "curl" or "luacurl" "libcurl".  A quick google search seemed to have a mixture of these
>
>
> ________________________________
> From: [hidden email] <[hidden email]> on behalf of Dirk Laurie <[hidden email]>
> Sent: 25 March 2019 13:34
> To: Lua mailing list
> Subject: Re: Problem trying to download a webpage
>
> I suppose this does not answer your question, but ... if you don't
> explicitly need SSL, you can get away with curl.
>
> I figured out the 'if' below a year or two ago, and now just
> cut-and-paste it every time.
>
> Having loaded lua-curl as "curl", this is all:
>
> logins = {}
> session = curl.easy()
> if curl.OPT_COOKIEFILE and curl.OPT_COOKIEJAR then
>     cookies = os.tmpname()
>     session:setopt(curl.OPT_COOKIEFILE,cookies)
>     session:setopt(curl.OPT_COOKIEJAR,cookies)
>     logins[session] = cookies
>   else
>     message("Your 'curl' does not support cookies. You will be anonymous.")
>   end
> session:setopt{url="https://www.attheraces.com/racecards",writefunction=print}:perform()
>
> I see the entire HTML code of the landing page.
>
> To store it in a table, just put something like 'function(str)
> table.insert(buffer,str) end' instead of 'print' having done
> 'buffer={}' first.
>
>
>
> Op Ma. 25 Mrt. 2019 om 13:51 het Geoff Smith <[hidden email]> geskryf:
> >
> > I am trying to download a webpage using Lua 5.1 and Luasec (not sure which version)
> >
> > I am seeing the same error on several pages, but one example is [[https://www.attheraces.com/racecards]]
> > I am getting an error stating "sslv3 alert handshake failure"
> >
> > I am not an expert on the https protocol so not sure what I need to do to fix this problem. I am assuming i need to tell it to use TLS v1.2 or later to make it work. Not proved that this is the problem though. Here is gist of my latest code attempt that still gives me the same error.
> >
> > ssl.newcontext({mode="client", protocol="tlsv1_2"})
> >
> > local url = [[https://www.attheraces.com/racecards]]
> >
> > webText, code, headers, status = https.request({url=url,  protocol="tlsv1_2"})
> > print(code, headers, status)
> >
> > I have tried protocol = "sslv23" and "sslv3"  neither fixed the problem
> >
> > Any help on this issue would be most appreciated, thanks.
> >
> >
> >
> >
> >
> >
> >
> > Virus-free. www.avast.com
>
>
> Virus-free. www.avast.com

Reply | Threaded
Open this post in threaded view
|

Re: Problem trying to download a webpage

aryajur
In reply to this post by Geoff Smith
I am using LuaSec 0.7 and Lua 5.3.4 and this works fine for me:

hs = require("ssl.https")
hs.request([[<a href="https://www.attheraces.com/racecards]">https://www.attheraces.com/racecards]])

On Mon, Mar 25, 2019 at 4:51 AM Geoff Smith <[hidden email]> wrote:
I am trying to download a webpage using Lua 5.1 and Luasec (not sure which version)

I am seeing the same error on several pages, but one example is [[https://www.attheraces.com/racecards]]
I am getting an error stating "sslv3 alert handshake failure"

I am not an expert on the https protocol so not sure what I need to do to fix this problem. I am assuming i need to tell it to use TLS v1.2 or later to make it work. Not proved that this is the problem though. Here is gist of my latest code attempt that still gives me the same error.

ssl.newcontext({mode="client", protocol="tlsv1_2"})


webText, code, headers, status = https.request({url=url,  protocol="tlsv1_2"})
print(code, headers, status)

I have tried protocol = "sslv23" and "sslv3"  neither fixed the problem

Any help on this issue would be most appreciated, thanks.







Virus-free. www.avast.com
Reply | Threaded
Open this post in threaded view
|

Re: Problem trying to download a webpage

Vadim A. Misbakh-Soloviov
In reply to this post by Dirk Laurie-2
> Unfortunatly I can't offer Windows-specific advice, but there must be
> many list members who can.
it should use same naming on all the platforms.
So, `curl = require"cURL"` should be fine.



Reply | Threaded
Open this post in threaded view
|

Re: Problem trying to download a webpage

Geoff Smith
Thanks for the help guys on this problem

milind's post was very interesting as the same identical code on my setup generates the error.

I have discovered I am using Luasec version 0.6a and Lua V5.1. I am assuming the lua version difference is not the significant factor. The fault i am seeing is most likely due to me not having luasec version 0.7.

I wasted 2 days last year on getting LuaRocks to work but could never figure out an unspecificed linker error it produced, so I now hate LuaRocks with a passion and cant/wont use it. 

Could anyone with Windows Binaries for Luasec v0.7 please post me a zip file here or point me at a download link for them ?

Thanks again for your help

Geoiff

From: [hidden email] <[hidden email]> on behalf of Vadim A. Misbakh-Soloviov <[hidden email]>
Sent: 26 March 2019 07:26
To: Lua mailing list
Subject: Re: Problem trying to download a webpage
 
> Unfortunatly I can't offer Windows-specific advice, but there must be
> many list members who can.
it should use same naming on all the platforms.
So, `curl = require"cURL"` should be fine.