Luasocket 2.0: socket.http.request using POST

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Luasocket 2.0: socket.http.request using POST

William Trenker
I've been having problems getting at least 2 web servers (Apache
mod-php, and lighttpd with fastcgi php support) to accept request data
using the POST method.

Consider (using Lua 5.1rc with LuaSockets 2.0 compiled in):

require'socket.http'
response_body = socket.http.request("http://mysite.com/test.php","a=1")
print(response_body)

with test.php on the server:

<?php
if (!isset($_POST['a'])) { echo "No a in POST data\n"; }
else { echo "a = ".$_POST['a']."\n"; }
?>

When run, the lua script prints "No a in POST data" indicating that
the "a=1" wasn't seen by the server.

The same thing happens when I use the table version:

require'socket.http'
require'ltn12'
response_body = {}
request_body = "a=1"
socket.http.request{
    url = "http://mysite.com/test.php",
    method = "POST",
    headers = {
         ["Content-Length"] = string.len(request_body)
     },
     source = ltn12.source.string(request_body),
     sink = ltn12.sink.table(response_body)
}
table.foreach(response_body,print)

Again I get "No a in POST data".

However, if I add the additional header:
         ["Content-Type"] =  "application/x-www-form-urlencoded"
then I get the output "a = 1" indicating the server has now seen the POST body.

Any thoughts?

Cheers,
Bill
Reply | Threaded
Open this post in threaded view
|

Re: Luasocket 2.0: socket.http.request using POST

Keith Howe
On 2/1/06, William Trenker <[hidden email]> wrote:
> However, if I add the additional header:
>          ["Content-Type"] =  "application/x-www-form-urlencoded"
> then I get the output "a = 1" indicating the server has now seen the POST body.
>
> Any thoughts?

I'm not exactly sure what the issue is here, as this seems correct to
me. Without the content-type specified, the web server (and therefore
PHP script) has no way to interpret the data you are sending in the
POST body, and its safest option is to simply ignore it entirely. It
is perfectly legal HTTP to send data in a POST body that is not
x-www-form-urlencoded, so this header is required if you expect it to
be interpreted as form data in label=value pairs.

I think the most likely source of confusion is that the PHP object is
calling itself $_POST, though what it really seems to be doing is
parsing form encoded post data, and does not handle arbitrary post
data. Of course, I know next to nothing about PHP :)

- Keith
Reply | Threaded
Open this post in threaded view
|

Re: Luasocket 2.0: socket.http.request using POST

William Trenker
On 2/1/06, Keith Howe <[hidden email]> wrote:

>  On 2/1/06, William Trenker <[hidden email]> wrote:
> > However, if I add the additional header:
> >          ["Content-Type"] =  "application/x-www-form-urlencoded"
> > then I get the output "a = 1" indicating the server has now seen the POST body.
> >
> > Any thoughts?
>
> I'm not exactly sure what the issue is here, as this seems correct to
> me. Without the content-type specified, the web server (and therefore
> PHP script) has no way to interpret the data you are sending in the
> POST body, and its safest option is to simply ignore it entirely. It
> is perfectly legal HTTP to send data in a POST body that is not
> x-www-form-urlencoded, so this header is required if you expect it to
> be interpreted as form data in label=value pairs.

 That's what I thought, too.  Yet I can't get this to work without the
"Content-Type" header.  A quick inspection of the Python urllib2.py
module (includes functionality similar to LuaSocket's
socket.http.request) seems to indicate that the Content-Type header is
needed:

         if req.has_data():
             data = req.get_data()
             h.putrequest('POST', req.get_selector())
             if not 'Content-type' in req.headers:
                 h.putheader('Content-type',
                             'application/x-www-form-urlencoded')
             if not 'Content-length' in req.headers:
                 h.putheader('Content-length', '%d' % len(data))
         else:
             h.putrequest('GET', req.get_selector())

 Bill
Reply | Threaded
Open this post in threaded view
|

Hello!

Are Leistad-2
Hello fellow Lua users!

This is a message of three purposes:

1) to see if posting to the list works

2) to thank the Lua team for the wonders of Lua!

3) to tell that Lua is being emdedded with great success
in yet another system

Regards,

  Are
--
Reply | Threaded
Open this post in threaded view
|

Re: Hello!

D Burgess-4
it works


> 1) to see if posting to the list works
>
Reply | Threaded
Open this post in threaded view
|

Re: Luasocket 2.0: socket.http.request using POST

Mark Edgar
In reply to this post by William Trenker
William Trenker wrote:

> On 2/1/06, Keith Howe <[hidden email]> wrote:
>
>> On 2/1/06, William Trenker <[hidden email]> wrote:
>>
>>>However, if I add the additional header:
>>>         ["Content-Type"] =  "application/x-www-form-urlencoded"
>>>then I get the output "a = 1" indicating the server has now seen the POST body.
>>>
>>>Any thoughts?
>>
>>I'm not exactly sure what the issue is here, as this seems correct to
>>me. Without the content-type specified, the web server (and therefore
>>PHP script) has no way to interpret the data you are sending in the
>>POST body, and its safest option is to simply ignore it entirely. It
>>is perfectly legal HTTP to send data in a POST body that is not
>>x-www-form-urlencoded, so this header is required if you expect it to
>>be interpreted as form data in label=value pairs.
>
>
>  That's what I thought, too.  Yet I can't get this to work without the
> "Content-Type" header.

Why would you expect your server's CGI implementation to recognize the
request body as application/x-www-form-urlencoded data in the absense of
  a content-type header which indicates this?  Why do you expect it to
work without the content-type header?  Has it worked this way on another
CGI implementation?  Do you have a browser which sends POST form data
without including the proper content-type header?

                                        -Mark
Reply | Threaded
Open this post in threaded view
|

Re: Luasocket 2.0: socket.http.request using POST

lego-2
On 2/2/06, Mark Edgar <[hidden email]> wrote:

> Why would you expect your server's CGI implementation to recognize the
> request body as application/x-www-form-urlencoded data in the absense of
>   a content-type header which indicates this?  Why do you expect it to
> work without the content-type header?  Has it worked this way on another
> CGI implementation?  Do you have a browser which sends POST form data
> without including the proper content-type header?

RFC 2616 (7.2.1 ) says that a given implementation SHOULD send a
Content-Type whenever there is a body, and that only if there is not
you MAY try to guess what's there.

So you SHOULD not expect it  but it MAY work.

Luis

--
This information is top security. When you have read it, destroy yourself.
-- Marshall McLuhan
Reply | Threaded
Open this post in threaded view
|

Re: Luasocket 2.0: socket.http.request using POST

William Trenker
On 2/2/06, LEGO <[hidden email]> wrote:
On 2/2/06, Mark Edgar <[hidden email]> wrote:

> Why would you expect your server's CGI implementation to recognize the
> request body as application/x-www-form-urlencoded data in the absense of
>   a content-type header which indicates this?  Why do you expect it to
> work without the content-type header?

 RFC 2616 (7.2.1 ) says that a given implementation SHOULD send a
Content-Type whenever there is a body, and that only if there is not
you MAY try to guess what's there.

Actually, I brought this subject up because the POST variant of the simple, automatic form of socket.http.request doesn't generate the Content-Type header. The simple form includes a 2nd string argument which signifies a POST request with the contents of the 2nd argument as the body of the request. The code in socket/http.lua automatically generates the necessary Content-Length header, but it doesn't generate a Content-Type header.  I suspect from the feedback to my post that the code should be fixed. I wanted to get some confirmation here before I posted a bug report. 

Thanks for everyone's input.

Bill

Reply | Threaded
Open this post in threaded view
|

Re: Luasocket 2.0: socket.http.request using POST

lego-2
On 2/7/06, William Trenker <[hidden email]> wrote:

> On 2/2/06, LEGO <[hidden email]> wrote:
> >  RFC 2616 (7.2.1 ) says that a given implementation SHOULD send a
> > Content-Type whenever there is a body, and that only if there is not
> > you MAY try to guess what's there.
>
>  Actually, I brought this subject up because the POST variant of the simple,
> automatic form of socket.http.request doesn't generate the Content-Type
> header. The simple form includes a 2nd string argument which signifies a
> POST request with the contents of the 2nd argument as the body of the
> request. The code in socket/http.lua automatically generates the necessary
> Content-Length header, but it doesn't generate a Content-Type header.  I
> suspect from the feedback to my post that the code should be fixed. I wanted
> to get some confirmation here before I posted a bug report.

IMO, unless there is a good reason to ignore it, ignoring a
recomendation's SHOULD should be considered a bug in an implementation
even if is not a MUST.

So again in my IMO it is good material for a bug report.

BTW, if you consider something a bug, just open a bug report (just
after checking it has not been reported already, as that would be
wasting other peoples time). If it is not a real bug it just took a
programmer 2-3 minutes to mark it as "not a bug" or "won't fix", if it
is actualy a bug then it is known which in most cases is half of the
solution. If it is an debatable thing (AKA opinion report) the very
reflection on the issue may give an offspring of its own which can be
a good thing.

Anyway, as a programmer I can tell you that even not having to do
anything to fix it, dismissing a bug report is always cause of
satisfaction :-).

--
This information is top security. When you have read it, destroy yourself.
-- Marshall McLuhan
Reply | Threaded
Open this post in threaded view
|

Re: Luasocket 2.0: socket.http.request using POST

Diego Nehab-3
Hi,

I wouldn't call this a bug. It is a feature request. A bug is something
that behaves differently from specification.

The simple API simply does not support the definition of a content-type
header (call it oversight).  The content-length header is a completely
different story. It *has* to be there. When I wrote the code, the most
popular servers did not accept accept chunked posts. Besides it is *not*
ambiguous at all. Content-type, on the other hand, is ambiguous.
Why should I define it as being application/x-www-form-urlencoded
instead of, say, multipart/mixed? Is it really more common?

Regards,
Diego.
Reply | Threaded
Open this post in threaded view
|

Re: Luasocket 2.0: socket.http.request using POST

lego-2
On 2/7/06, Diego Nehab <[hidden email]> wrote:
> Hi,
>
> I wouldn't call this a bug. It is a feature request. A bug is something
> that behaves differently from specification.
I agree that a SHOULD is not a MUST but still believe that SHOULDs
should be implemented unless there is a specific reason not to, but
then again it's just my personal opinion, it is juat a SHOULD not a
MUST.


> The simple API simply does not support the definition of a content-type
> header (call it oversight).
> The content-length header is a completely
> different story. It *has* to be there.
for compatibility with HTTP/1.0 yes it has to, but in HTTP/1.1 it does
not not necesarily have to be there, in some  are cases in it MIGHT or
MUST NOT not be there (4.4 explains when) and the RFC implicitly tells
that in some parts it  should not be used.

RFC 2116 is somewhat hard to understand, It MAY not be ambiguous, but
it SHOULD certailnly be clearer :-).

But it in 14.13 it clearly says
"
  ... In HTTP, it  SHOULD be sent whenever the message's length can be
determined prior
   to being transferred, unless this is prohibited by the rules in
   section 4.4.
"

> When I wrote the code, the most
> popular servers did not accept accept chunked posts. Besides it is *not*
> ambiguous at all. Content-type, on the other hand, is ambiguous.

I think it SHOULD give a Content-Type: so the server knows what's
there, and does not need to guess. But that's my personal opinion.

Then RFC2616 tells in p3.7
"
   Note that some older HTTP applications do not recognize media type
   parameters. When sending data to older HTTP applications,
   implementations SHOULD only use media type parameters when they are
   required by that type/subtype definition.
"

> Why should I define it as being application/x-www-form-urlencoded
> instead of, say, multipart/mixed? Is it really more common?

That's a Presentation layer issue not certainly a Session layer one so
HTTP (been L5 protocol) is not realy involved with that. I would say
let's see if MIME tells us what to do but guess... p 19.4.1 tells that
"HTTP is not a MIME-compliant protocol" so in our case it does not
apply.

So that's preety much an implementors issue, many ITU-T standards
define the interoperation between layes the HTTP/1.1 RFC does not. It
does not specify what SAP (service access points) one module has to
provide to another one and how that interface is proved.

In the HTTP field there is the CGI specification (
http://hoohoo.ncsa.uiuc.edu/cgi/ ) that provides that but it does not
apply to this case. So you are free to implement it as you want.

I would use application/x-www-form-urlencoded as it's more common
unless you want to transfer files in which case RFC1867 might be more
apopriate.

But anyway I still believe (an Opinion Report if you want to consider
it that way) that a Content-Type should be provided.

Luis.
--
This information is top security. When you have read it, destroy yourself.
-- Marshall McLuhan
Reply | Threaded
Open this post in threaded view
|

Re: Luasocket 2.0: socket.http.request using POST

Petite Abeille
In reply to this post by Diego Nehab-3

On Feb 07, 2006, at 17:19, Diego Nehab wrote:

> Why should I define it as being application/x-www-form-urlencoded
> instead of, say, multipart/mixed? Is it really more common?

Not sure about "multipart/mixed" in the context of a POST, but
"multipart/form-data" is indeed popular :)

http://www.faqs.org/rfcs/rfc2388.html

Cheers

--
PA, Onnay Equitursay
http://alt.textdrive.com/

Reply | Threaded
Open this post in threaded view
|

Re: Luasocket 2.0: socket.http.request using POST

Javier Guerra Giraldez
In reply to this post by lego-2
On Tuesday 07 February 2006 12:39 pm, LEGO wrote:
> > Why should I define it as being application/x-www-form-urlencoded
> > instead of, say, multipart/mixed? Is it really more common?
>
> That's a Presentation layer issue not certainly a Session layer one so
> HTTP (been L5 protocol) is not realy involved with that. I would say

please don't mix that OSI crap with real-world networking!

--
Javier

attachment0 (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Luasocket 2.0: socket.http.request using POST

Diego Nehab-3
In reply to this post by lego-2
Hi,

>> I wouldn't call this a bug. It is a feature request. A bug is something
>> that behaves differently from specification.
> I agree that a SHOULD is not a MUST but still believe that SHOULDs
> should be implemented unless there is a specific reason not to, but
> then again it's just my personal opinion, it is juat a SHOULD not a
> MUST.

I gave the reason: http.request does not know the content-type of the
request body. Only the programmer knowns. Therefore, LuaSocket doesn't
send the header by magic.

Either I change the simple API (thus making it complicated), or I just
drop it.  I will probably drop the simple API for POST. Whoever is
coding a POST might as well just write his own wrapper over the complex
API and send the header.

> for compatibility with HTTP/1.0 yes it has to, but in HTTP/1.1 it does
> not not necesarily have to be there, in some  are cases in it MIGHT or
> MUST NOT not be there (4.4 explains when) and the RFC implicitly tells
> that in some parts it  should not be used.

Once you read the RFC for the 23213123'rd time, you forget the wording
and remember only the reasoning behiind things.

The server needs to know where the request body ends. HTTP uses one of
three ways to define the length of a message: connection close,
content-type, or chunked.  In the case of a POST, the client
can't close the connection, because it wants to read the response. It
can't send chunked, because most servers won't accept it. By exclusion,
the content-length has to be there.

Try passing a source as the body to the complex http.request function
and try not giving a content-length field. You will see I try sending
chunked. At this point, there is nothing else I can do.

In the simple API, I get a string. I have its length. So it's a piece of
cake: send content-length. Servers will be happy all over the place.

> But anyway I still believe (an Opinion Report if you want to consider
> it that way) that a Content-Type should be provided.

Nobody is questioning this fact. What I am saying is that it's not up to
LuaSocket to add this header behind your back. It is up to the LuaSocket
user to add the correct header.

Regards,
Diego.