luasocket/soap: can I keep the connection open?

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

luasocket/soap: can I keep the connection open?

Norbert Kiesel
Hi,

I use luasoap to talk to a server (vmware) and right now every soap call
creates a new TCP socket (and closes it after the call).  As I'm actually
using https, this means setting up the ssl connection for every call.  I
thought I might avoid that by explicitly opening and closing the socket
(in my login/logout methods), but I can't figure out how to do that.

Any ideas/hints?

Best,
  Norbert



Reply | Threaded
Open this post in threaded view
|

Re: luasocket/soap: can I keep the connection open?

Scott Vokes
On Sat, Oct 2, 2010 at 9:56 PM,  <[hidden email]> wrote:
> I use luasoap to talk to a server (vmware) and right now every soap call
> creates a new TCP socket (and closes it after the call).  As I'm actually
> using https, this means setting up the ssl connection for every call.  I
> thought I might avoid that by explicitly opening and closing the socket
> (in my login/logout methods), but I can't figure out how to do that.
   Perhaps I'm misunderstanding you, but you should be able to use
socket.setoption(yoursocket, 'keepalive') to maintain a pool of active
connections and socket.close(yoursocket) to explictly close them.

Scott

Reply | Threaded
Open this post in threaded view
|

Re: luasocket/soap: can I keep the connection open?

Norbert Kiesel
On Sun, 2010-10-03 at 09:11 -0400, Scott Vokes wrote:
> On Sat, Oct 2, 2010 at 9:56 PM,  <[hidden email]> wrote:
> > I use luasoap to talk to a server (vmware) and right now every soap call
> > creates a new TCP socket (and closes it after the call).  As I'm actually
> > using https, this means setting up the ssl connection for every call.  I
> > thought I might avoid that by explicitly opening and closing the socket
> > (in my login/logout methods), but I can't figure out how to do that.
>    Perhaps I'm misunderstanding you, but you should be able to use
> socket.setoption(yoursocket, 'keepalive') to maintain a pool of active
> connections and socket.close(yoursocket) to explictly close them.

Tried that, but does not make a (visible) difference.  The socket is
closed before the return of the http.request call done in the soap
library.

I also disabled all obvious close methods in soap.lua, socket.lua, and
http.lua (just to see which one closes it, I know that is not a real
solution), but socket is still closed (e.g. getfd() on the socket
returns -1 after the http.request call).


</nk>



Reply | Threaded
Open this post in threaded view
|

Re: luasocket/soap: can I keep the connection open?

Petite Abeille

On Oct 3, 2010, at 7:25 PM, Norbert Kiesel wrote:

> Tried that, but does not make a (visible) difference.  The socket is
> closed before the return of the http.request call done in the soap
> library.

At first glance, luasocket's http module doesn't support persistent connection.

http.lua:215 adjustheaders(reqt): ["connection"] = "close, TE",
http.lua:327 trequest(reqt): h:close()




Reply | Threaded
Open this post in threaded view
|

Re: luasocket/soap: can I keep the connection open?

Norbert Kiesel
On Sun, 2010-10-03 at 19:41 +0200, Petite Abeille wrote:

> On Oct 3, 2010, at 7:25 PM, Norbert Kiesel wrote:
>
> > Tried that, but does not make a (visible) difference.  The socket is
> > closed before the return of the http.request call done in the soap
> > library.
>
> At first glance, luasocket's http module doesn't support persistent connection.
>
> http.lua:215 adjustheaders(reqt): ["connection"] = "close, TE",
> http.lua:327 trequest(reqt): h:close()

Yeah, but even after commenting out the h:close() the socket is  still
closed somehow.

</nk>



Reply | Threaded
Open this post in threaded view
|

Re: luasocket/soap: can I keep the connection open?

Petite Abeille-2-2

On Oct 3, 2010, at 8:04 PM, Norbert Kiesel wrote:

>> At first glance, luasocket's http module doesn't support persistent connection.
>>
>> http.lua:215 adjustheaders(reqt): ["connection"] = "close, TE",
>> http.lua:327 trequest(reqt): h:close()
>
> Yeah, but even after commenting out the h:close() the socket is  still
> closed somehow.

By default, the request is asking for the connection to be closed by the server (http.lua:215), so perhaps this is not unexpected, no?


Reply | Threaded
Open this post in threaded view
|

Re: luasocket/soap: can I keep the connection open?

Norbert Kiesel
On Sun, 2010-10-03 at 20:09 +0200, Petite Abeille wrote:

> On Oct 3, 2010, at 8:04 PM, Norbert Kiesel wrote:
>
> >> At first glance, luasocket's http module doesn't support persistent connection.
> >>
> >> http.lua:215 adjustheaders(reqt): ["connection"] = "close, TE",
> >> http.lua:327 trequest(reqt): h:close()
> >
> > Yeah, but even after commenting out the h:close() the socket is  still
> > closed somehow.
>
> By default, the request is asking for the connection to be closed by the server (http.lua:215), so perhaps this is not unexpected, no?

First of all thanks for your help.
I removed the "close" from that line (and also the TE in another
attempt), but that does not make a difference.  I even ran it under gdb
and put breakpoints on meth_close and meth_shutdown (the C functions
implementing socket:close() and socket:shutdown(), but they are not hit
and still getfd() returns -1 after the request.

</nk>



Reply | Threaded
Open this post in threaded view
|

Re: luasocket/soap: can I keep the connection open?

BogdanM
On Sun, Oct 3, 2010 at 9:26 PM, Norbert Kiesel <[hidden email]> wrote:
On Sun, 2010-10-03 at 20:09 +0200, Petite Abeille wrote:
> On Oct 3, 2010, at 8:04 PM, Norbert Kiesel wrote:
>
> >> At first glance, luasocket's http module doesn't support persistent connection.
> >>
> >> http.lua:215 adjustheaders(reqt): ["connection"] = "close, TE",
 

</nk>



> >> http.lua:327 trequest(reqt): h:close()
> >
> > Yeah, but even after commenting out the h:close() the socket is  still
> > closed somehow.
>
> By default, the request is asking for the connection to be closed by the server (http.lua:215), so perhaps this is not unexpected, no?

First of all thanks for your help.
I removed the "close" from that line (and also the TE in another
attempt), but that does not make a difference.  I even ran it under gdb
and put breakpoints on meth_close and meth_shutdown (the C functions
implementing socket:close() and socket:shutdown(), but they are not hit
and still getfd() returns -1 after the request.

The HTTP keepalive mechanism isn't supposed to keep connections opened for too long:

http://en.wikipedia.org/wiki/Keepalive

Best,
Bogdan

Reply | Threaded
Open this post in threaded view
|

Re: luasocket/soap: can I keep the connection open?

Petite Abeille
In reply to this post by Norbert Kiesel

On Oct 3, 2010, at 8:26 PM, Norbert Kiesel wrote:

> First of all thanks for your help.
> I removed the "close" from that line (and also the TE in another
> attempt), but that does not make a difference.  I even ran it under gdb
> and put breakpoints on meth_close and meth_shutdown (the C functions
> implementing socket:close() and socket:shutdown(), but they are not hit
> and still getfd() returns -1 after the request.

The thing is... LuaSocket http module doesn't support persistent connection by default... so you will have to tweak it a bit...

Now... from an HTTP point of view... you need the client side request to be HTTP/1.1 or use connection = keep-alive for earlier version.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html

And have the HTTP server honor such persistent request... which the server could choose not to.

The easiest way to check what's going on would be to record what HTTP request header are emitted and what response header are returned.


Reply | Threaded
Open this post in threaded view
|

Re: luasocket/soap: can I keep the connection open?

Norbert Kiesel
On Sun, 2010-10-03 at 20:38 +0200, Petite Abeille wrote:

> On Oct 3, 2010, at 8:26 PM, Norbert Kiesel wrote:
>
> > First of all thanks for your help.
> > I removed the "close" from that line (and also the TE in another
> > attempt), but that does not make a difference.  I even ran it under gdb
> > and put breakpoints on meth_close and meth_shutdown (the C functions
> > implementing socket:close() and socket:shutdown(), but they are not hit
> > and still getfd() returns -1 after the request.
>
> The thing is... LuaSocket http module doesn't support persistent connection by default... so you will have to tweak it a bit...
>
> Now... from an HTTP point of view... you need the client side request to be HTTP/1.1 or use connection = keep-alive for earlier version.
>
> http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html
>
> And have the HTTP server honor such persistent request... which the server could choose not to.
>
> The easiest way to check what's going on would be to record what HTTP request header are emitted and what response header are returned.

I added print's at various places and I see a "HTTP/1.1 200 OK" status
coming back from the server (print before the return in
metat.__index:receivestatusline).  And the headers sent are
{host=192.168.6.185,content-type=text/xml,te=trailers,
user-agent=LuaSocket2.0.2,content-length=326,
cookie=vmware_soap_session="52530721-2986-0459-f803-481be1afe990",
soapaction="urn:vim25/4.1"}

I think the socket/http.lua already uses mode = "keep-open" because I
have a content-length in my header.

I could try to switch the server from HTTPS to HTTP so that I could
record the conversation with wireshark, but ultimately I have to support
SOAP over HTTPS.

</nk>



Reply | Threaded
Open this post in threaded view
|

Re: luasocket/soap: can I keep the connection open?

Diego Nehab-3
Hi,

I have not tried to implement support for persistent connections in
LuaSocket because of the complications involved. I don't know how you
modified the API to support multiple requests per connection, so I
can't tell where it is going wrong. But if you use http.request(), it
will close the connection as soon as it done getting the response
body.

Let me know if you need any help.

Regards,
Diego

Reply | Threaded
Open this post in threaded view
|

Re: luasocket/soap: can I keep the connection open?

Norbert Kiesel
On Sun, 2010-10-03 at 22:28 -0300, Diego Nehab wrote:
> Hi,
>
> I have not tried to implement support for persistent connections in
> LuaSocket because of the complications involved. I don't know how you
> modified the API to support multiple requests per connection, so I
> can't tell where it is going wrong. But if you use http.request(), it
> will close the connection as soon as it done getting the response
> body.
>
I use luasoap with luasocket to send soap calls to VMware servers.  This
overall works beautifully and functionality wise I'm a happy camper
(having a wsdl->lua converter would be nice though, but that's a
different topic).

However, for even simple tasks I end up doing dozens of SOAP calls,
ending up sometimes doing about 1000 calls within 5 minutes.  And all
that over HTTPS (I added a little piece of code to the open method to
call ssl.wrap on the socket). Right now even a simple call takes about
0.2 seconds, so I'm reaching the end of the line and I'm looking for
speedups.  Trying to re-use the connection instead of re-creating it for
every call was/is one of my ideas.

Just to see if that would work, I tried to simply comment out all
"close" calls I could find (in socke.lua and http.lua) to see if the
server side would accepts the 2nd call over the same connection.  But
somehow I'm missing one place, because the socket is still closed after
the first request and I can't figure out where/how that happens.

</nk>



Reply | Threaded
Open this post in threaded view
|

Re: luasocket/soap: can I keep the connection open?

Matthew Wild
On 4 October 2010 02:55, Norbert Kiesel <[hidden email]> wrote:

> On Sun, 2010-10-03 at 22:28 -0300, Diego Nehab wrote:
>> Hi,
>>
>> I have not tried to implement support for persistent connections in
>> LuaSocket because of the complications involved. I don't know how you
>> modified the API to support multiple requests per connection, so I
>> can't tell where it is going wrong. But if you use http.request(), it
>> will close the connection as soon as it done getting the response
>> body.
>>
> I use luasoap with luasocket to send soap calls to VMware servers.  This
> overall works beautifully and functionality wise I'm a happy camper
> (having a wsdl->lua converter would be nice though, but that's a
> different topic).
>
> However, for even simple tasks I end up doing dozens of SOAP calls,
> ending up sometimes doing about 1000 calls within 5 minutes.  And all
> that over HTTPS (I added a little piece of code to the open method to
> call ssl.wrap on the socket). Right now even a simple call takes about
> 0.2 seconds, so I'm reaching the end of the line and I'm looking for
> speedups.  Trying to re-use the connection instead of re-creating it for
> every call was/is one of my ideas.
>
> Just to see if that would work, I tried to simply comment out all
> "close" calls I could find (in socke.lua and http.lua) to see if the
> server side would accepts the 2nd call over the same connection.  But
> somehow I'm missing one place, because the socket is still closed after
> the first request and I can't figure out where/how that happens.
>

As has been said earlier in the thread, even if you remove all close()
calls the server can still close the connection after sending the
response, and it probably does. Unless the server and client both
support HTTP 1.1 persistent connections then expect the connection to
close.

As Diego said, LuaSocket doesn't support HTTP 1.1 persistent
connections, and I don't know a Lua HTTP library that does (not that I
can claim to have exhaustively searched). If you're sure your server
supports them then it may be worth implementing.

Alternatively I have a HTTP client library that might be easier to
adapt. I've not tested it with persistent connections, but it may be
easier to extend to do so. The library is part of Prosody, but I have
a project here that pulls the HTTP library into something standalone:
http://code.matthewwild.co.uk/lahttp (it uses Squish to pull in
http://hg.prosody.im/trunk/file/4495403470cb/net/http.lua and a couple
of others).

It's on my todo to implement myself at some point, but I don't have
time right now.

Regards,
Matthew

PS. Is the RPC server a part of VMware? I'm wondering if HTTP is
actually requirement... :)

Reply | Threaded
Open this post in threaded view
|

Re: luasocket/soap: can I keep the connection open?

Diego Nehab-3
In reply to this post by Norbert Kiesel
Hi Norbert,

> Just to see if that would work, I tried to simply comment out all
> "close" calls I could find (in socke.lua and http.lua) to see if the
> server side would accepts the 2nd call over the same connection.  But
> somehow I'm missing one place, because the socket is still closed after
> the first request and I can't figure out where/how that happens.

Hmmm. Did you try taking a look at the C code?
Have you ruled out the server closing the connection?

Regards,
Diego

Reply | Threaded
Open this post in threaded view
|

Re: luasocket/soap: can I keep the connection open?

Daurnimator
In reply to this post by Matthew Wild
On 4 October 2010 23:43, Matthew Wild <[hidden email]> wrote:

> As Diego said, LuaSocket doesn't support HTTP 1.1 persistent
> connections, and I don't know a Lua HTTP library that does (not that I
> can claim to have exhaustively searched). If you're sure your server
> supports them then it may be worth implementing.
>
> Regards,
> Matthew
>
> PS. Is the RPC server a part of VMware? I'm wondering if HTTP is
> actually requirement... :)
>
>

I wrote a HTTP 1.1 lua client at one point (to test my HTTP 1.1
server): it was actually quite a simple task. I can't find it now, but
it should take a good coder 15mins to bash it out (use some utility
functions from luasocket)

Daurn.

Reply | Threaded
Open this post in threaded view
|

Re: luasocket/soap: can I keep the connection open?

Norbert Kiesel
On Tue, 2010-10-05 at 09:45 +1100, Quae Quack wrote:

> On 4 October 2010 23:43, Matthew Wild <[hidden email]> wrote:
> > As Diego said, LuaSocket doesn't support HTTP 1.1 persistent
> > connections, and I don't know a Lua HTTP library that does (not that I
> > can claim to have exhaustively searched). If you're sure your server
> > supports them then it may be worth implementing.
> >
> > Regards,
> > Matthew
> >
> > PS. Is the RPC server a part of VMware? I'm wondering if HTTP is
> > actually requirement... :)
> >
> >
>
> I wrote a HTTP 1.1 lua client at one point (to test my HTTP 1.1
> server): it was actually quite a simple task. I can't find it now, but
> it should take a good coder 15mins to bash it out (use some utility
> functions from luasocket)

I guess that makes me a bad coder (or at least not a good one) because I
looked at socket.lua and socket/http.lua for more than 15 minutes and
could not figure it out.  I guess I could write a primitive HTTP 1.1
client in that time, but not with all the nice error handling etc. that
socket/http.lua comes with.  If you find your code, I'd be grateful if
you could share it.

</nk>



Reply | Threaded
Open this post in threaded view
|

Re: luasocket/soap: can I keep the connection open?

Norbert Kiesel
In reply to this post by Diego Nehab-3
On Mon, 2010-10-04 at 19:10 -0300, Diego Nehab wrote:

> Hi Norbert,
>
> > Just to see if that would work, I tried to simply comment out all
> > "close" calls I could find (in socke.lua and http.lua) to see if the
> > server side would accepts the 2nd call over the same connection.  But
> > somehow I'm missing one place, because the socket is still closed after
> > the first request and I can't figure out where/how that happens.
>
> Hmmm. Did you try taking a look at the C code?
> Have you ruled out the server closing the connection?

I looked at the C code but as I said earlier even when commenting out
all the close() calls I could find getfd() still returned -1.  I could
see the server closing it's side, but that should still leave the local
fd unchanged (though of course no longer usable if the server really
closed the connection), no?

</nk>





Reply | Threaded
Open this post in threaded view
|

Re: luasocket/soap: can I keep the connection open?

Diego Nehab-3
Hi,

The problem with persistent connections is that your code has to be
robust to the server sending 100 status or not, and closing the
connection on you or not. I am sure different servers will disrespect
the standard in different ways, which means more debugging. You also
need to create an API that hides this complexity from the user in some
way.

It would have taken me a lot more than 15 minutes to do it, so it
would be great if Mr. Quack could find that piece of code... :)

> I looked at the C code but as I said earlier even when commenting out
> all the close() calls I could find getfd() still returned -1.  I could
> see the server closing it's side, but that should still leave the local
> fd unchanged (though of course no longer usable if the server really
> closed the connection), no?

If getfd() is returning -1 then somewhere some C code wrote -1 to the socket.
Look for SOCKET_INVALID. I would have happened either in usocket.c or
in wsocket.c. Somebody is calling socket_destroy() on this socket.

Regards,
Diego

PS: On the other hand, if the server is closing the connection, then
you have other problems to solve as well.

Reply | Threaded
Open this post in threaded view
|

Re: luasocket/soap: can I keep the connection open?

Pierre Chapuis
 On Tue, 5 Oct 2010 01:27:37 -0300, Diego Nehab
 <[hidden email]> wrote:

> The problem with persistent connections is that your code has to be
> robust to the server sending 100 status or not, and closing the
> connection on you or not. I am sure different servers will disrespect
> the standard in different ways, which means more debugging. You also
> need to create an API that hides this complexity from the user in
> some
> way.
>
> It would have taken me a lot more than 15 minutes to do it, so it
> would be great if Mr. Quack could find that piece of code... :)

 I think I've been bitten by that too when trying to implement HTTP
 Digest Authentication (client-side) in Lua on top of LuaSocket.

 I think the server I was using wants the first two requests in the
 same connection (the first one being the one that is answered to
 with a 401 status).

 Anyway, I'd be happy to get that piece of code too!

--
 Pierre Chapuis