[OT?] What does Firefox do that lua-curl can't?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[OT?] What does Firefox do that lua-curl can't?

Dirk Laurie-2
There is a package on Luarocks called lua-curl. With it, I can surf
the Internet form inside my Lua program.

Or so I thought. After all, all that Firefox and friends do is to send
a URL with everything you are communicating to a server and displaying
or otherwise utilizing the stuff it sends back.

I found that I could do general surfing thta way but did not succeed
in logging in to a site on which I am a registered user.

Maybe this is off-topic for this list. But maybe someone has used
lua-curl and knows what to do.

Reply | Threaded
Open this post in threaded view
|

Re: [OT?] What does Firefox do that lua-curl can't?

Viacheslav Usov
On Wed, Jun 27, 2018 at 6:10 PM Dirk Laurie <[hidden email]> wrote:

> Maybe this is off-topic for this list. But maybe someone has used lua-curl and knows what to do.

One problem that I am aware of is that it uses OpenSSL for SSL/TLS communications, which is pretty standard these days at least where credentials are exchanged, and OpenSSL needs to have "trusted root authority" certificates to validate certificate trust chains that are involved in that. It does not have any built into it, and while the OpenSSL C-level API lets the user supply those in various forms, the Lua wrapper is less flexible. I think you can still provide a file system path to load them from, and you can disable that altogether, but the colleague who knows the gory details has called it a day.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: [OT?] What does Firefox do that lua-curl can't?

Daniel Silverstone
In reply to this post by Dirk Laurie-2
On Wed, Jun 27, 2018 at 18:10:11 +0200, Dirk Laurie wrote:
> I found that I could do general surfing thta way but did not succeed
> in logging in to a site on which I am a registered user.
>
> Maybe this is off-topic for this list. But maybe someone has used
> lua-curl and knows what to do.

Possibly a lack of cookie handling?  Javascript on the page for captchas?  Lots
of possibilities.

D.

--
Daniel Silverstone                         http://www.digital-scurf.org/
PGP mail accepted and encouraged.            Key Id: 3CCE BABE 206C 3B69

Reply | Threaded
Open this post in threaded view
|

Re: [OT?] What does Firefox do that lua-curl can't?

Ryan
Essentially the question you're asking is "can I make a web browser in Lua" to which the answer is absolutely.  However, you would need to include (or, if you were feeling especially ambitious, write) a Javascript interpreter so that you could use sites that use Javascript.

On Wed, Jun 27, 2018, 12:20 PM Daniel Silverstone <[hidden email]> wrote:
On Wed, Jun 27, 2018 at 18:10:11 +0200, Dirk Laurie wrote:
> I found that I could do general surfing thta way but did not succeed
> in logging in to a site on which I am a registered user.
>
> Maybe this is off-topic for this list. But maybe someone has used
> lua-curl and knows what to do.

Possibly a lack of cookie handling?  Javascript on the page for captchas?  Lots
of possibilities.

D.

--
Daniel Silverstone                         http://www.digital-scurf.org/
PGP mail accepted and encouraged.            Key Id: 3CCE BABE 206C 3B69

Reply | Threaded
Open this post in threaded view
|

Re: [OT?] What does Firefox do that lua-curl can't?

Jeff Pohlmeyer
In reply to this post by Dirk Laurie-2
Dirk Laurie wrote:
> There is a package on Luarocks called lua-curl. With it, I can surf
> the Internet form inside my Lua program.
> ...
> I found that I could do general surfing thta way but did not succeed
> in logging in to a site on which I am a registered user.

This page has a few tips:
https://curl.haxx.se/docs/httpscripting.html

This might also be interesting:
https://luakit.github.io/

- Jeff

Reply | Threaded
Open this post in threaded view
|

Re: [OT?] What does Firefox do that lua-curl can't?

Niccolo Medici
In reply to this post by Dirk Laurie-2
On 6/27/18, Dirk Laurie <[hidden email]> wrote:
> There is a package on Luarocks called lua-curl. With it, I can surf
> the Internet form inside my Lua program.
>
> Or so I thought. [...] I could do general surfing thta way but did not succeed
> in logging in to a site on which I am a registered user.


Sadly, nowadays you may not even be able to build a web scrapper using
just an http library.

Many mainstream sites build their content with JavaScript (long gone
are the days of "progressive enhancement / graceful degradation"). A
JavaScript engine is not enough: you need to implement all the APIs of
the browser (like "local storage").

Reply | Threaded
Open this post in threaded view
|

Re: [OT?] What does Firefox do that lua-curl can't?

szbnwer@gmail.com
hi there! :)

+1 is the user agent string that can vary some sites

you can use the inspector of any browser and mimic cookies headers and
post+get fileds, and u have to get the same result, however there are
much stuffs that are loaded after the initial page that you are
requesting for... like just try to use umatrix ff plugin, it will
wreck a lotsa pages cuz of the external ajax from domains of google
that is blocked by default...

i have a future plan to make ast from pages, track down function calls
for those parts of any requests that are not relevant, and write some
lua code for them. however there can be shared data with other
function chains on the road, so its even harder cuz those... its also
necessary to track changes on a site, so multiple loads of
same/similar pages can basically help, and then i can initially mark
changing parts, and every other change stops the execution and telling
me that something possibly important have changed, so modifications
maybe required in my code... but this way everything can be used on a
semi-automatic way without overhead and with huge freedom to do
whatever, like combining data from multiple pages, automatize, throw
away trash, extend functionality, make custom displays and whatever
without the need of reimplementing the functionality of the whole
frontend world, and without running unknown stuffs from the web in the
era of proproetary bloatwares and meltown, spectre rowhammer and
whatever... :D anyway check out greasemonkey if you wouldnt know about
it, its a similar tool, but still requires a full browser...

btw luakit is really kawl, and evolving fast! give it a try, but only
from source, repositories around are much likely very outdated, or
this was the case in the near past... webkit have a debug api that
make u able to manipulate whatever from an external app, see the
brackets editor for further hints about the debug api :)

have fun, and all the bests for everyone around! :)

Reply | Threaded
Open this post in threaded view
|

Re: [OT?] What does Firefox do that lua-curl can't?

aryajur
In reply to this post by Dirk Laurie-2
You need a cookie management module like this:

It can be used something like this:

        lc = require("luacookie")
local cm = lc.new()
-- Login 
local hdrs,msg,resp, URI
print("Try connection to "..URI)
-- Got the form fill details from the chrome developer console
local b,c,l,h = http.request(URI ,formencode({
go = "ok",
nexturl = [[http://www.website.com/]] ,
username = [[username]] ,
password = [[La8TZmMc]] ,
autolog = "X"
}))
if not b then
print("Error making connection: "..tostring(c))
return
end
hdrs,msg = cm.add(l,URI)
local ft = {}
print("Try the check login url with the cookie.")
URI = l.location
hdrs, msg = cm.getCookieHeader(URI)
if not hdrs then return nil,msg end
resp = {http.request{
url = URI,
sink = ltn12.sink.table(ft),
headers = hdrs
}
}
print("Done with the http request.")
if not resp[1] then 
return nil,resp[2]
end


So basically you create a new cookie manager (cm) and then after each request just pass the header table to the add function in the cookie manager and then in the next request get the header using getCookieHeader function of the cookie manager. This has worked for me in the past few occasions requiring cookie management for websites that require login.



On Wed, Jun 27, 2018, 9:10 AM Dirk Laurie <[hidden email]> wrote:
There is a package on Luarocks called lua-curl. With it, I can surf
the Internet form inside my Lua program.

Or so I thought. After all, all that Firefox and friends do is to send
a URL with everything you are communicating to a server and displaying
or otherwise utilizing the stuff it sends back.

I found that I could do general surfing thta way but did not succeed
in logging in to a site on which I am a registered user.

Maybe this is off-topic for this list. But maybe someone has used
lua-curl and knows what to do.