LuaSocket - how can a tcp client detect a closed connection?

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

LuaSocket - how can a tcp client detect a closed connection?

Romulo Bahiense
Hi,

I'm attempting to implement a Keep-Alive connection using the socket.http library. I'm already able to detect and handle the details (http headers, persisting the socket userdata etc.)

The problem is: how to detect when the server (httpd: apache+fastcgi) closes the connection? I've tried to watch the results of client:send() while sending the reqline and detecting if the error is "closed", but it seems that the client is acting as if nothing had happened until it tries to read from the same socket.

Another solution I've tried is to socket.protect socket.http.request{} and repeat the whole request when the connection was closed, but sometimes the server _do_ close the connection and repeating the request would be wrong.

Am I doing something wrong? Any ideas?

WinXP, Lua 5.0.2, LuaSocket-2.0.1-lua50-win32.

Thanks.


Reply | Threaded
Open this post in threaded view
|

Re: LuaSocket - how can a tcp client detect a closed connection?

Javier Guerra Giraldez
On Thursday 01 March 2007, Romulo Bahiense wrote:
> The problem is: how to detect when the server (httpd: apache+fastcgi)
> closes the connection? I've tried to watch the results of client:send()
> while sending the reqline and detecting if the error is "closed", but it
> seems that the client is acting as if nothing had happened until it
> tries to read from the same socket.

this is the expected behaviour, by definition calling close() on a TCP stream 
only means "i won't write anymore data to this stream".  it's perfectly valid 
to keep reading for as long as you want.

so, if the server closes the stream you can still send data; and if it calls 
read(), will get it.  only when the second part (the client, in your case) 
calls close() too, is the whole stream finished.

now, should it be detectable by the application?... i have no idea


-- 
Javier

Attachment: pgph02epcOZJr.pgp
Description: PGP signature

Reply | Threaded
Open this post in threaded view
|

Re: LuaSocket - how can a tcp client detect a closed connection?

Sam Roberts-2
In reply to this post by Romulo Bahiense
On Thu, Mar 01, 2007 at 06:57:50PM -0300, Romulo Bahiense wrote:
> I'm attempting to implement a Keep-Alive connection using the 
> socket.http library. I'm already able to detect and handle the details 
> (http headers, persisting the socket userdata etc.)

> The problem is: how to detect when the server (httpd: apache+fastcgi) 
> closes the connection? I've tried to watch the results of client:send() 
> while sending the reqline and detecting if the error is "closed", but it 
> seems that the client is acting as if nothing had happened until it 
> tries to read from the same socket.

Maybe the http server hasn't closed it's read side of the connection,
only the write side?

Clients are supposed to monitor the read side of the tcp link
continuously, that allows the server to close at any time or to send a
failure response before an entire request is sent by the client.

I don't know how you'd handle that with luasocket's http, but it might
be something to look into.

Sam


Reply | Threaded
Open this post in threaded view
|

Re: LuaSocket - how can a tcp client detect a closed connection?

D Burgess-4
In reply to this post by Javier Guerra Giraldez
On 3/2/07, Javier Guerra <[hidden email]> wrote:
so, if the server closes the stream you can still send data; and if it calls
read(), will get it.  only when the second part (the client, in your case)
calls close() too, is the whole stream finished.

Not so on Windows

Reply | Threaded
Open this post in threaded view
|

Re: LuaSocket - how can a tcp client detect a closed connection?

D Burgess-4
In reply to this post by Romulo Bahiense
Maybe try a  select().
select() should return an error FDS when doing a send()

db

On 3/2/07, Romulo Bahiense <[hidden email]> wrote:
Hi,

I'm attempting to implement a Keep-Alive connection using the
socket.http library. I'm already able to detect and handle the details
(http headers, persisting the socket userdata etc.)

The problem is: how to detect when the server (httpd: apache+fastcgi)
closes the connection? I've tried to watch the results of client:send()
while sending the reqline and detecting if the error is "closed", but it
seems that the client is acting as if nothing had happened until it
tries to read from the same socket.

Another solution I've tried is to socket.protect socket.http.request{}
and repeat the whole request when the connection was closed, but
sometimes the server _do_ close the connection and repeating the request
would be wrong.

Am I doing something wrong? Any ideas?

WinXP, Lua 5.0.2, LuaSocket-2.0.1-lua50-win32.

Thanks.



Reply | Threaded
Open this post in threaded view
|

Re: LuaSocket - how can a tcp client detect a closed connection?

Sam Roberts-2
In reply to this post by D Burgess-4
On Fri, Mar 02, 2007 at 09:40:57AM +1100, David Burgess wrote:
> On 3/2/07, Javier Guerra <[hidden email]> wrote:
> >so, if the server closes the stream you can still send data; and if it 
> >calls
> >read(), will get it.  only when the second part (the client, in your case)
> >calls close() too, is the whole stream finished.
> 
> Not so on Windows

Or on unix. We must use shutdown() to do a half-close, which is what
well behaved clients and servers do.

close() does a full-close, shuts the read and write side. Same on
windows, AFAIK:

  http://tangentsoft.net/wskfaq/newbie.html

  2.10

Which doesn't mean everybody does it correctly!

Sam


Reply | Threaded
Open this post in threaded view
|

Re: LuaSocket - how can a tcp client detect a closed connection?

Taj Khattra
Or on unix. We must use shutdown() to do a half-close, which is what
well behaved clients and servers do.

<rant> http://cr.yp.to/tcpip/twofd.html </rant>

Reply | Threaded
Open this post in threaded view
|

Re: LuaSocket - how can a tcp client detect a closed connection?

Sam Roberts-2
On Thu, Mar 01, 2007 at 03:21:09PM -0800, Taj Khattra wrote:
> >Or on unix. We must use shutdown() to do a half-close, which is what
> >well behaved clients and servers do.
> 
> <rant> http://cr.yp.to/tcpip/twofd.html </rant>

Do you understand what he's trying to say? I don't. Half-close is needed
only when you read AND write on the same fd, as for HTTP. In his
pipeline case, each side of the connection only reads OR writes, so
doing a close() works fine, you can do as he describes without trouble.

Maybe he had trouble on one of the ancient systems he alleges can be
crashed by calling getsockopt(), and hasn't got around to updating the
web page.

Anyhow, representing a socket as a single fd allows it to be used by
existing code that reads and writes from a single fd, think "file" here,
avoiding "device specific garbage", as he says. Does he want two fds for
files, as well, one reading one writing? Calling two fds "obvious" is a
bit of a stretch... :-)

Sam


Reply | Threaded
Open this post in threaded view
|

Re: LuaSocket - how can a tcp client detect a closed connection?

Taj Khattra
On 3/1/07, Sam Roberts <[hidden email]> wrote:
doing a close() works fine, you can do as he describes without trouble.

nope, it doesn't work.

here's a (useless) pipeline:
   % echo howdy | cat | cat
   howdy
   %

works as expected.  now let's make the middle cat a tcp service:
   % tcpserver -RHl0 0 6969 sh -c 'exec cat' &

let's try our pipeline again, this time using the cat tcp service:
   % tcpclient -RHl0 0 6969 sh -c ' echo howdy >&7 & exec cat <&6'
   howdy
   <waiting... still waiting... hmm... looks like we are stuck>

here comes the "device specific garbage" to the rescue :), where the
"device" in this case is tcp:
   % echo 'main() { shutdown(1, 1); }' >tcpfin.c
   % cc -o tcpfin tcpfin.c
   % tcpclient -RHl0 0 6969 sh -c '(echo howdy; ./tcpfin) >&7 & exec cat <&6'
   howdy
   %

works, but it's unfortunate that we have to stick that tcp specific
crud in there.  having separate read and write ofiles, just like pipes
do, would obviate the crud.

to stay at least partly on-topic for lua-l:  it seems like i can't
write tcpfin using luasocket, which just doesn't seem right...

Reply | Threaded
Open this post in threaded view
|

Re: LuaSocket - how can a tcp client detect a closed connection?

Sam Roberts-2
On Fri, Mar 02, 2007 at 12:47:36AM -0800, Taj Khattra wrote:
> On 3/1/07, Sam Roberts <[hidden email]> wrote:
> >doing a close() works fine, you can do as he describes without trouble.
> 
> nope, it doesn't work.

Ah, I missed the "doesn't work with ucspi" part.

> here's a (useless) pipeline:
>    % echo howdy | cat | cat
>    howdy

There are TWO pipes here

> works as expected.  now let's make the middle cat a tcp service:
>    % tcpserver -RHl0 0 6969 sh -c 'exec cat' &

> let's try our pipeline again, this time using the cat tcp service:
>    % tcpclient -RHl0 0 6969 sh -c ' echo howdy >&7 & exec cat <&6'
>    howdy

and only ONE tcp connection here. To make this the same, you would need
a hypothetical utility that worked like:

% tcp --read 9002 "cat"
% tcp --read 9001 --write localhost:9002 "cat"
% tcp --write localhost:9001

This would result in:

  echo howdy {tcp link} cat {tcplink} cat


Or use ssh:

sroberts@pebble:~% echo howdy | /usr/bin/ssh root@192.168.130.11 cat | cat
howdy
sroberts@pebble:~%

DJB wishes sockets were defined like unix pipes, unidirectional, but
instead they are designed like unix ttys and unix files, bidirectional.
Making them more like pipes would make them less like files and ttys, so
ucspi would work work better, and many other things would work worse.

Making things more uniform in unix would take a lot more work than a
tweak to the BSD socket API, and then it wouldn't be unix, it would be
plan9.

> to stay at least partly on-topic for lua-l:  it seems like i can't
> write tcpfin using luasocket, which just doesn't seem right...

I haven't used luasocket, but

http://www.cs.princeton.edu/~diego/professional/luasocket/tcp.html#shutdown

doesn't work as advertised?

Sam


Reply | Threaded
Open this post in threaded view
|

Re: LuaSocket - how can a tcp client detect a closed connection?

Taj Khattra
There are TWO pipes here

yes, a pipe is unidirectional, so i need two.

and only ONE tcp connection here.

yes, a tcp connection is bidirectional, so i need just one.

Or use ssh:

i can't see why ssh decided to make an appearance here, unless it's
specific to the example i gave.  there's no tcp ofile being shared in
your pipeline, so it doesn't address the "TCP FIN after last writer
leaves" issue.  or are you suggesting that a tcp socket should always
be restricted to a single local process (in this case, ssh)?

DJB wishes sockets were defined like unix pipes, unidirectional, but
instead they are designed like unix ttys and unix files, bidirectional.

it's about readers and writers, not unidirectional vs. bidirectional.

a pipe has separate read and write fds because it has readers and
writers, not because it's unidirectional.  the kernel knows when the
last writer leaves, and can subsequently signal an eof to the readers
(and vice versa, when the last reader leaves).  a tty/file doesn't
have the same requirement.

a tcp socket also has readers and writers, and the kernel should,
without being prodded, signal an eof (TCP FIN to its tcp peer) after
the last local writer leaves.  seems more like a pipe than a tty/file.

and yes, i know it's too late to change the socket api now and am not
suggesting that we go and break all the code out there for the sake of
"uniformity".

I haven't used luasocket, but

http://www.cs.princeton.edu/~diego/professional/luasocket/tcp.html#shutdown

doesn't work as advertised?

sure it does.  but i can't also shutdown() an arbitrary fd.  since
luasocket already has a binding to shutdown() i was hoping that i
could.  but it seems like the low-level binding isn't being exposed.
or maybe i didn't look close enough.

Reply | Threaded
Open this post in threaded view
|

Re: LuaSocket - how can a tcp client detect a closed connection?

Dave Dodge
On Sat, Mar 03, 2007 at 06:31:37AM -0800, Taj Khattra wrote:
> >There are TWO pipes here
> 
> yes, a pipe is unidirectional, so i need two.
> 
> >and only ONE tcp connection here.
> 
> yes, a tcp connection is bidirectional, so i need just one.

No, you _need_ two if you want to compare them meaningfully to
traditional Unix pipes.  By making use of bidirectional communication
through a socket, you are taking advantage of an additional feature
that sockets offer over pipes, and you should not be surprised that
using this extra functionality breaks the similarity with the behavior
of a pipe.

If you want sockets to behave like pipes, then use them
unidirectionally like pipes.  And if you want pipes to behave like
sockets, you can get something pretty close by using socketpair()
instead of pipe() to create them.

It should be noted that in SVR4 UNIX, regular pipes actually _are_
bidirectional.  For example all pipes on Solaris are bidirectional,
and on IRIX you can request them at link time.  Of course just as with
sockets, any application making use of this bidirectionality will need
to be redesigned if you want to run it on a system with the more
traditional unidirectional pipes.

                                                  -Dave Dodge

Reply | Threaded
Open this post in threaded view
|

Re: LuaSocket - how can a tcp client detect a closed connection?

gary ng
--- Dave Dodge <[hidden email]> wrote:
> Of course just as with sockets, any application 
> making use of this bidirectionality will need
> to be redesigned if you want to run it on a system
> with the more traditional unidirectional pipes.
> 
I have a question. If I dup2() the stdin/stdout(as in
inetd/tcpserver) using one end of socketpair() and
execute the child, would that make it work even the
child is expecting pipe like fd as its stdin/stdout ?

If that is the case, would that be a better option
than pipe(), as it has bidirectional capability yet
can be used as unidirectional pipe ?

Another related issue, must I use shutdown() for
socket or just close(fd) would do the same thing ?



 
____________________________________________________________________________________
Food fight? Enjoy some healthy debate 
in the Yahoo! Answers Food & Drink Q&A.
http://answers.yahoo.com/dir/?link=list&sid=396545367

Reply | Threaded
Open this post in threaded view
|

Re: LuaSocket - how can a tcp client detect a closed connection?

Gé Weijers
--- Dave Dodge <[hidden email]> wrote:
Of course just as with sockets, any application 
making use of this bidirectionality will need
to be redesigned if you want to run it on a system
with the more traditional unidirectional pipes.

I have a question. If I dup2() the stdin/stdout(as in
inetd/tcpserver) using one end of socketpair() and
execute the child, would that make it work even the
child is expecting pipe like fd as its stdin/stdout ?

Using dup2 would give you two read/write file descriptors. If you close one nothing happens, because the Unix kernel waits until the last once is closed before doing anything. Closing stdout/fd 1 in a inetd client will not magically do a 'shutdown' on the sending side of a TCP connection. Bernstein's problem still exists.


Another related issue, must I use shutdown() for
socket or just close(fd) would do the same thing ?


A call to 'shutdown' signals that you are done sending or receiving information. It's mostly useful if you want to effectively send an end-of-file but you want to continue reading the reply. If you want to stop reading and writing altogether just use 'close', it effectively does a shutdown in both directions and remove the socket handle.


--
Gé Weijers



Reply | Threaded
Open this post in threaded view
|

Re: LuaSocket - how can a tcp client detect a closed connection?

gary ng
--- Gé Weijers <[hidden email]> wrote:
> Using dup2 would give you two read/write file
> descriptors. If you  
> close one nothing happens, because the Unix kernel
> waits until the  
> last once is closed before doing anything. Closing
> stdout/fd 1 in a  
> inetd client will not magically do a 'shutdown' on
> the sending side  
> of a TCP connection. Bernstein's problem still
> exists.
But if I instead do a shutdown(), the receiving
end(i.e. the one I fork() with a dup2 stdin/out, both
use the same underlying socket), it would solve the
problem ? As it seems that shutdown a socket would
make all fds that relies on it signal EOF ?




 
____________________________________________________________________________________
It's here! Your new message!  
Get new email alerts with the free Yahoo! Toolbar.
http://tools.search.yahoo.com/toolbar/features/mail/

Reply | Threaded
Open this post in threaded view
|

Re: LuaSocket - how can a tcp client detect a closed connection?

Gé Weijers

On Mar 5, 2007, at 9:38 AM, gary ng wrote:


--- Gé Weijers <[hidden email]> wrote:
Using dup2 would give you two read/write file
descriptors. If you  
close one nothing happens, because the Unix kernel
waits until the  
last once is closed before doing anything. Closing
stdout/fd 1 in a  
inetd client will not magically do a 'shutdown' on
the sending side  
of a TCP connection. Bernstein's problem still
exists.
But if I instead do a shutdown(), the receiving
end(i.e. the one I fork() with a dup2 stdin/out, both
use the same underlying socket), it would solve the
problem ? As it seems that shutdown a socket would
make all fds that relies on it signal EOF ?


A TCP stream contains 2 logically separate streams, one going out and once coming in. Shutdown allows you to end the conversation on one of them, but keep the connection open in the other direction. If you do a fd:shutdown("send") the other side of the connection gets a 0 byte read. fd:read will still work until the other side does a shutdown or close.

So if you send some information you'd get code like:

fd:write(data)
fd:shutdown("send")
....
fd:read()

This will allow you to read incoming data after you have sent all there is to send.

Doing a 'dup2' creates another file descriptor, but this descriptor does not differ in any way from the original one. The underlying software (e.g. the network stack or file system) will only be notified of a 'close' when all file descriptors pointing to the same file/socket have been closed. So 'shutdown' is still necessary.

Daniel Bernstein's observation was that if the socket interfaces would have returned 2 file descriptors instead of one 'close' could have done the job, and software could have been oblivious of the differences between pipes, files and sockets. He's not alone in thinking that. See Rob Pike's presentation: http://herpolhode.com/rob/ugly.pdf.

I'm not so sure than porting the BSD socket interface to all kinds of other OSes and languages was such a good idea.



--
Gé Weijers



Reply | Threaded
Open this post in threaded view
|

Re: LuaSocket - how can a tcp client detect a closed connection?

gary ng
--- Gé Weijers <[hidden email]> wrote:
> Daniel Bernstein's observation was that if the
> socket interfaces would have returned 2 file 
> descriptors instead of one 'close' could  
> have done the job
However, given the generic and widespread usage of
fork()/exec(), even 2 file descriptors may not be
enough. As they are in general passed to childs which
unless explicitly closed in the forked child, would
have the same effect as it is now. So a shutdown() is
still needed which effectively means, close this
pipe/socket regardless of how many fd are dup().

I was in this situation trying to create a loopback
pipe in lua like pipethrough("gzip -dc"). Initially, I
juse use pipe(2) create the pipe pair then dup
stdin/stdout and fork()/exec(). However, even I close
the write end of the parent, I still don't get the
result because it was passed to the fork() child. I
have to explicitly close them before the exec().

With socketpair(), a shutdown("write") from the parent
would do the job.



 
____________________________________________________________________________________
Need Mail bonding?
Go to the Yahoo! Mail Q&A for great tips from Yahoo! Answers users.
http://answers.yahoo.com/dir/?link=list&sid=396546091

Reply | Threaded
Open this post in threaded view
|

Re: LuaSocket - how can a tcp client detect a closed connection?

Mark Edgar-2
On 3/5/07, gary ng <[hidden email]> wrote:
However, given the generic and widespread usage of
fork()/exec(), even 2 file descriptors may not be
enough. As they are in general passed to childs which
unless explicitly closed in the forked child, would
have the same effect as it is now. So a shutdown() is
still needed which effectively means, close this
pipe/socket regardless of how many fd are dup().

Neglecting to explicitly (or implicity via FD_CLOEXEC) close unused
descriptors is a bug in the program, not a flaw in the design of
POSIX.

shutdown() is not legal for use with pipes.  It is specific to sockets.

In the hypothetical design of dual-fd sockets (let's call these
twockets), a close() on the writing twocket is equivalent to
shutdown(sock, SHUT_RD) on the socket, and a close() on the reading
twocket is equivalent to shutdown(sock, SHUT_WR) on the socket.

I was in this situation trying to create a loopback
pipe in lua like pipethrough("gzip -dc"). Initially, I
juse use pipe(2) create the pipe pair then dup
stdin/stdout and fork()/exec(). However, even I close
the write end of the parent, I still don't get the
result because it was passed to the fork() child. I
have to explicitly close them before the exec().

Right.  You need to ensure that each unused end is closed before
calling exec, or use FD_CLOEXEC instead to accomplish this.

BTW, the "ex" API can do this for you in "pure" Lua.  See the popen2()
example at http://lua-users.org/wiki/ExtensionProposal

    -Mark

Reply | Threaded
Open this post in threaded view
|

Re: LuaSocket - how can a tcp client detect a closed connection?

Mark Edgar-2
On 3/6/07, Mark Edgar <[hidden email]> wrote:
In the hypothetical design of dual-fd sockets (let's call these
twockets), a close() on the writing twocket is equivalent to
shutdown(sock, SHUT_RD) on the socket, and a close() on the reading
twocket is equivalent to shutdown(sock, SHUT_WR) on the socket.

Oops, I think I meant that SHUT_RD is equivalent to closing the
reading twocket, and SHUT_WR is equivalent to closing the writing
twocket.

    -Mark

Reply | Threaded
Open this post in threaded view
|

Re: LuaSocket - how can a tcp client detect a closed connection?

gary ng
In reply to this post by Mark Edgar-2
--- Mark Edgar <[hidden email]> wrote:

> On 3/5/07, gary ng <[hidden email]> wrote:
> > However, given the generic and widespread usage of
> > fork()/exec(), even 2 file descriptors may not be
> > enough. As they are in general passed to childs
> which
> > unless explicitly closed in the forked child,
> would
> > have the same effect as it is now. So a shutdown()
> is
> > still needed which effectively means, close this
> > pipe/socket regardless of how many fd are dup().
> 
> Neglecting to explicitly (or implicity via
> FD_CLOEXEC) close unused
> descriptors is a bug in the program, not a flaw in
> the design of
> POSIX.
> 
If it is just exec(), this may be true. But what if I
just need to fork() and run seperately for something
else(like spawn and wait or just running other things
in seperate process space) ? I would still need to
pick and close things to prevent this situation. It is
easy to just identify the two handles involved(like
the popen2 example) but when there are more forking
than a simple one to one(like parent multiple child), 
it can become complicated. As the parent pipe with one
child may leak to other child during forking().

I ended up doing a loop like this immediately after
fork():

for i=2,256 do close(i) end

i.e. only limit the communication using 0/1 bewteen
parent/child.

Sure it is a bug(as anything not function as expected
is) but it is still complication. .



 
____________________________________________________________________________________
Need Mail bonding?
Go to the Yahoo! Mail Q&A for great tips from Yahoo! Answers users.
http://answers.yahoo.com/dir/?link=list&sid=396546091