luaL_loadbuffer API cannot deal with shebang

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

luaL_loadbuffer API cannot deal with shebang

Baozeng
Hello all,
My last two emails are not showed normally. Sorry that I have to sent again.
I have a simple Lua file, for example, hello.lua. The contents of the file starts with a shebang
 I first read the file content into a buffer. And then use luaL_loadbuffer and lua_pcall to call the function. However, I found that luaL_loadbuffer cannot return normally because it does not deal with the buffer started with a shebang. Any suggestion. Thanks.

--
     Best Regards,
     Baozeng Ding
                                                                
Reply | Threaded
Open this post in threaded view
|

Re: luaL_loadbuffer API cannot deal with shebang

Jonathan Goble
On Tue, May 21, 2019 at 10:15 PM Baozeng <[hidden email]> wrote:
>
> Hello all,
> My last two emails are not showed normally. Sorry that I have to sent again.

Your emails are showing up fine for me.

Reply | Threaded
Open this post in threaded view
|

Re: luaL_loadbuffer API cannot deal with shebang

Francisco Olarte
In reply to this post by Baozeng
Baozeng:

On Wed, May 22, 2019 at 4:15 AM Baozeng <[hidden email]> wrote:
> I have a simple Lua file, for example, hello.lua. The contents of the file starts with a shebang

This is not a simple lua file, shebangs are not valid lua.

>  I first read the file content into a buffer. And then use luaL_loadbuffer and lua_pcall to call the function. However, I found that luaL_loadbuffer cannot return normally because it does not deal with the buffer started with a shebang.

If you take a look at the sources of luaL_loadfile() you'll notice it
skips initial garbage ( BOMs, shell comments ) with an auxiliary
skipcomment() function.

You have two simple solutions:

1.- If you can, use luaL_loadfile instead of loading yourself, let
auxlib do the work.

2.- Otherwise, reimplement ( by creative imitation if you want ) skip
comment to skip the shebang lines on the buffer ( test * buf, if it's
# increment it until you hit end of buffer decrementing sz along the
way, the usual stuff ) ( I would wrap this in a
load_buffer_skip_shebang ).

Francisco Olarte.

Reply | Threaded
Open this post in threaded view
|

Re: luaL_loadbuffer API cannot deal with shebang

Coda Highland


On Wed, May 22, 2019 at 4:20 AM Francisco Olarte <[hidden email]> wrote:
Baozeng:

On Wed, May 22, 2019 at 4:15 AM Baozeng <[hidden email]> wrote:
> I have a simple Lua file, for example, hello.lua. The contents of the file starts with a shebang

This is not a simple lua file, shebangs are not valid lua.

>  I first read the file content into a buffer. And then use luaL_loadbuffer and lua_pcall to call the function. However, I found that luaL_loadbuffer cannot return normally because it does not deal with the buffer started with a shebang.

If you take a look at the sources of luaL_loadfile() you'll notice it
skips initial garbage ( BOMs, shell comments ) with an auxiliary
skipcomment() function.

You have two simple solutions:

1.- If you can, use luaL_loadfile instead of loading yourself, let
auxlib do the work.

2.- Otherwise, reimplement ( by creative imitation if you want ) skip
comment to skip the shebang lines on the buffer ( test * buf, if it's
# increment it until you hit end of buffer decrementing sz along the
way, the usual stuff ) ( I would wrap this in a
load_buffer_skip_shebang ).

Francisco Olarte.


Option 3: if (buffer[0] == '#' && buffer[1] == '!') { buffer[0] = '-'; buffer[1] = '-'; }

A shebang is syntactically a comment in shell scripting, so replacing one comment syntax with another is a fairly elegant solution IMO. It's an extremely cheap check, an extremely cheap patch, and completely safe.

/s/ Adam
Reply | Threaded
Open this post in threaded view
|

Re: luaL_loadbuffer API cannot deal with shebang

Dirk Laurie-2
Op Wo. 22 Mei 2019 om 15:41 het Coda Highland <[hidden email]> geskryf:

> On Wed, May 22, 2019 at 4:20 AM Francisco Olarte <[hidden email]> wrote:
>>
>> Baozeng:
>>
>> On Wed, May 22, 2019 at 4:15 AM Baozeng <[hidden email]> wrote:
>> > I have a simple Lua file, for example, hello.lua. The contents of the file starts with a shebang
>>
>> This is not a simple lua file, shebangs are not valid lua.

They're not in the language, but the manual does document that the
standalone interpreter skips such a line at the top of a file. So they
are in practice standard Lua unless you are using your own
incompatible custom interpreter.

Reply | Threaded
Open this post in threaded view
|

Re: luaL_loadbuffer API cannot deal with shebang

Roberto Ierusalimschy
> Op Wo. 22 Mei 2019 om 15:41 het Coda Highland <[hidden email]> geskryf:
>
> > On Wed, May 22, 2019 at 4:20 AM Francisco Olarte <[hidden email]> wrote:
> >>
> >> Baozeng:
> >>
> >> On Wed, May 22, 2019 at 4:15 AM Baozeng <[hidden email]> wrote:
> >> > I have a simple Lua file, for example, hello.lua. The contents of the file starts with a shebang
> >>
> >> This is not a simple lua file, shebangs are not valid lua.
>
> They're not in the language, but the manual does document that the
> standalone interpreter skips such a line at the top of a file. So they
> are in practice standard Lua unless you are using your own
> incompatible custom interpreter.

As already pointed out, as it is not part of the language,
luaL_readbuffer (and loadstring in Lua) does not handle it.
Only loadfile does that. For strings, if needed, it is an easy
task (both in C and in Lua) to skip that line if needed.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: luaL_loadbuffer API cannot deal with shebang

Francisco Olarte
In reply to this post by Dirk Laurie-2
Dirk:

On Wed, May 22, 2019 at 3:55 PM Dirk Laurie <[hidden email]> wrote:
> Op Wo. 22 Mei 2019 om 15:41 het Coda Highland <[hidden email]> geskryf:
...>> On Wed, May 22, 2019 at 4:15 AM Baozeng <[hidden email]> wrote:
> >> > I have a simple Lua file, for example, hello.lua. The contents of the file starts with a shebang
> >> This is not a simple lua file, shebangs are not valid lua.

> They're not in the language, but the manual does document that the
> standalone interpreter skips such a line at the top of a file. So they
> are in practice standard Lua unless you are using your own
> incompatible custom interpreter.

You are right. In fact, once you've pointed out I reread section 7,
and found it speaks of lines starting with "#", and the same
documented for loadfile. WHich explains why I din't find it in my
first manual / source reading as I was searching for '#!' ( did not
want to search for # in normal lua docs, too many #t find ).

Anyway, I was trying to point that he did not have a "simple lua
file", but a "lua script". Although I haven't tested what mechanism
require() and friends use...source seems to indicate loadfile.

Francisco Olarte.

Jim
Reply | Threaded
Open this post in threaded view
|

Re: luaL_loadbuffer API cannot deal with shebang

Jim
In reply to this post by Baozeng
On 5/22/19, Baozeng <[hidden email]> wrote:
> I have a simple Lua file, for example, hello.lua. The contents of the file
> starts with a shebang
>  I first read the file content into a buffer. And then use luaL_loadbuffer
> and lua_pcall to call the function. However, I found that luaL_loadbuffer
> cannot return normally because it does not deal with the buffer started
> with a shebang.

how about
int r = luaL_dofile ( ... ) ;
??

the result should be contained in the variable "r" now.
when loading the file content into a buffer you should save the
start of the second line in a pointer if the file starts with a shebang
("#!")  and then call load_string/buffer from that advanced position
via the new pointer value instead of the original start position.
this is due to Lua not using "#" to indicate comments.

Jim
Reply | Threaded
Open this post in threaded view
|

Re: luaL_loadbuffer API cannot deal with shebang

Jim
In reply to this post by Coda Highland
On 5/22/19, Coda Highland <[hidden email]> wrote:
> Option 3:
> if (buffer[0] == '#' && buffer[1] == '!')
> { buffer[0] = '-'; buffer[1] = '-'; }
>
> A shebang is syntactically a comment in shell scripting, so replacing one
> comment syntax with another is a fairly elegant solution IMO. It's an
> extremely cheap check, an extremely cheap patch, and completely safe.

indeed.
i also did that a while ago when i wrote a custom interpreter and totally
forgot about it.

for such cases a C API lua_main() function resembling the "Tcl_Main()"
Tcl C API procedure could be added. a simple interpreter could
then be implemented using this function (if said function does not use
readline, else even the standard interpreter could use it).

Reply | Threaded
Open this post in threaded view
|

Re: luaL_loadbuffer API cannot deal with shebang

Baozeng
In reply to this post by Jim


Jim <[hidden email]> 于2019年5月23日周四 上午4:12写道:
On 5/22/19, Baozeng <[hidden email]> wrote:
> I have a simple Lua file, for example, hello.lua. The contents of the file
> starts with a shebang
>  I first read the file content into a buffer. And then use luaL_loadbuffer
> and lua_pcall to call the function. However, I found that luaL_loadbuffer
> cannot return normally because it does not deal with the buffer started
> with a shebang.

how about
int r = luaL_dofile ( ... ) ;
??


LuaL_dofile is ok. But sometimes users may use luaL_loadbuffer to load a buffer that from 
a file.(There are many users, you cannot decide what they do).  So luaL_loadbuffer should have
the same behavior with luaL_loadfile.
the result should be contained in the variable "r" now.
when loading the file content into a buffer you should save the
start of the second line in a pointer if the file starts with a shebang
("#!")  and then call load_string/buffer from that advanced position
via the new pointer value instead of the original start position.
this is due to Lua not using "#" to indicate comments.



--
     Best Regards,
     Baozeng Ding
                                                                
Reply | Threaded
Open this post in threaded view
|

Re: luaL_loadbuffer API cannot deal with shebang

Baozeng
In reply to this post by Coda Highland


Coda Highland <[hidden email]> 于2019年5月22日周三 下午9:41写道:


On Wed, May 22, 2019 at 4:20 AM Francisco Olarte <[hidden email]> wrote:
Baozeng:

On Wed, May 22, 2019 at 4:15 AM Baozeng <[hidden email]> wrote:
> I have a simple Lua file, for example, hello.lua. The contents of the file starts with a shebang

This is not a simple lua file, shebangs are not valid lua.

>  I first read the file content into a buffer. And then use luaL_loadbuffer and lua_pcall to call the function. However, I found that luaL_loadbuffer cannot return normally because it does not deal with the buffer started with a shebang.

If you take a look at the sources of luaL_loadfile() you'll notice it
skips initial garbage ( BOMs, shell comments ) with an auxiliary
skipcomment() function.

You have two simple solutions:

1.- If you can, use luaL_loadfile instead of loading yourself, let
auxlib do the work.

2.- Otherwise, reimplement ( by creative imitation if you want ) skip
comment to skip the shebang lines on the buffer ( test * buf, if it's
# increment it until you hit end of buffer decrementing sz along the
way, the usual stuff ) ( I would wrap this in a
load_buffer_skip_shebang ).

Francisco Olarte.


Option 3: if (buffer[0] == '#' && buffer[1] == '!') { buffer[0] = '-'; buffer[1] = '-'; }

A shebang is syntactically a comment in shell scripting, so replacing one comment syntax with another is a fairly elegant solution IMO. It's an extremely cheap check, an extremely cheap patch, and completely safe.
This is a very good idea. I like it!  

/s/ Adam


--
     Best Regards,
     Baozeng Ding
                                                                
Reply | Threaded
Open this post in threaded view
|

Re: luaL_loadbuffer API cannot deal with shebang

Baozeng


Baozeng <[hidden email]> 于2019年5月23日周四 上午10:20写道:


Coda Highland <[hidden email]> 于2019年5月22日周三 下午9:41写道:


On Wed, May 22, 2019 at 4:20 AM Francisco Olarte <[hidden email]> wrote:
Baozeng:

On Wed, May 22, 2019 at 4:15 AM Baozeng <[hidden email]> wrote:
> I have a simple Lua file, for example, hello.lua. The contents of the file starts with a shebang

This is not a simple lua file, shebangs are not valid lua.

>  I first read the file content into a buffer. And then use luaL_loadbuffer and lua_pcall to call the function. However, I found that luaL_loadbuffer cannot return normally because it does not deal with the buffer started with a shebang.

If you take a look at the sources of luaL_loadfile() you'll notice it
skips initial garbage ( BOMs, shell comments ) with an auxiliary
skipcomment() function.

You have two simple solutions:

1.- If you can, use luaL_loadfile instead of loading yourself, let
auxlib do the work.

2.- Otherwise, reimplement ( by creative imitation if you want ) skip
comment to skip the shebang lines on the buffer ( test * buf, if it's
# increment it until you hit end of buffer decrementing sz along the
way, the usual stuff ) ( I would wrap this in a
load_buffer_skip_shebang ).

Francisco Olarte.


Option 3: if (buffer[0] == '#' && buffer[1] == '!') { buffer[0] = '-'; buffer[1] = '-'; }

A shebang is syntactically a comment in shell scripting, so replacing one comment syntax with another is a fairly elegant solution IMO. It's an extremely cheap check, an extremely cheap patch, and completely safe.
This is a very good idea. I like it!  

Hmm, the buffer is a const char *, so its content cannot be changed. A little pity. 

/s/ Adam


--
     Best Regards,
     Baozeng Ding
                                                                


--
     Best Regards,
     Baozeng Ding
                                                                
Reply | Threaded
Open this post in threaded view
|

Re: luaL_loadbuffer API cannot deal with shebang

Andrew Gierth
In reply to this post by Baozeng
>>>>> "Baozeng" == Baozeng  <[hidden email]> writes:

 Baozeng> LuaL_dofile is ok. But sometimes users may use luaL_loadbuffer
 Baozeng> to load a buffer that from a file.(There are many users, you
 Baozeng> cannot decide what they do). So luaL_loadbuffer should have
 Baozeng> the same behavior with luaL_loadfile.

But luaL_loadbuffer can't know whether the buffer actually came from a
file, and whether it was from the start of a file. So it makes more
sense for loadbuffer to accept only valid lua chunks, and require that
the caller handle any such details. (The other thing that dofile checks
that loadbuffer does not is whether the file starts with a BOM.)

--
Andrew.

Reply | Threaded
Open this post in threaded view
|

Re: luaL_loadbuffer API cannot deal with shebang

Baozeng


Andrew Gierth <[hidden email]> 于2019年5月23日周四 下午12:11写道:
>>>>> "Baozeng" == Baozeng  <[hidden email]> writes:

 Baozeng> LuaL_dofile is ok. But sometimes users may use luaL_loadbuffer
 Baozeng> to load a buffer that from a file.(There are many users, you
 Baozeng> cannot decide what they do). So luaL_loadbuffer should have
 Baozeng> the same behavior with luaL_loadfile.

But luaL_loadbuffer can't know whether the buffer actually came from a
file, and whether it was from the start of a file. So it makes more
sense for loadbuffer to accept only valid lua chunks, and require that
the caller handle any such details. (The other thing that dofile checks
that loadbuffer does not is whether the file starts with a BOM.)

--
Andrew.

Could we add some check in luaL_loadbuffer like the following?


  size_t ignore_sz = 3;

  if (size >= 2 && buff[0] == '#' && buff[1] == '!') {

     while(ignore_sz <= size && buff[ignore_sz-1] != '\n') {

      ignore_sz ++;

     }

     if(ignore_sz >= size) return -1;

     else {

       size = size - ignore_sz;

       buff = buff + ignore_sz;

     }

  }


--
     Best Regards,
     Baozeng Ding
                                                                
Reply | Threaded
Open this post in threaded view
|

Re: luaL_loadbuffer API cannot deal with shebang

Francisco Olarte
Baozeng:

On Thu, May 23, 2019 at 6:46 AM Baozeng <[hidden email]> wrote:

> Could we add some check in luaL_loadbuffer like the following?
>   size_t ignore_sz = 3;
>   if (size >= 2 && buff[0] == '#' && buff[1] == '!') {
>      while(ignore_sz <= size && buff[ignore_sz-1] != '\n') {
>       ignore_sz ++;
>      }
>      if(ignore_sz >= size) return -1;
>      else {
>        size = size - ignore_sz;
>        buff = buff + ignore_sz;
>      }
>   }

I would not do that, it could break some things, and doing the same
thing as load_file, which skips a potential BOM and a 1st line
STARTING WITH A SINGLE '#' ( not with a shebang, it skips every "shell
like comment" ) is pretty easy with a wrapper:
/** Could be better named loadbufferx_like_loadfile_does ;-> */
int luaL_loadbufferx_skip (lua_State *L,
               const char *buff,
               size_t sz,
               const char *name,
               const char *mode)
{
  const char BOM[]={0xEF,0xBB,0xBF};
  if (sz>=sizeof(BOM) && memcmp(buff, p, sizeof(BOM))==0) {
    buf+=sizeof(BOM);
    sz+=sizeof(BOM);
  }
  if (sz>0 && *buff=='#') {
    do {
      --sz;
      ++buf;
    } while(sz>0 && *buff!='\n');
    if (sz>0) { /* exit due to newline. */
      --sz;
      ++buf;
    }
  }
  /* let base function decide what to do with empty buffer if sz==0. */
  return luaL_loadbufferx(L, buff,sz, name,mode);
}
( UNTESTED, JUST A DRAFT ).

Francisco Olarte.

Reply | Threaded
Open this post in threaded view
|

Re: luaL_loadbuffer API cannot deal with shebang

Tim Hill


On May 23, 2019, at 2:25 AM, Francisco Olarte <[hidden email]> wrote:

 if (sz>=sizeof(BOM) && memcmp(buff, p, sizeof(BOM))==0) {
   buf+=sizeof(BOM);
   sz+=sizeof(BOM);
 }
 if (sz

Of course you mean:

sz -= sizeof(BOM);

—Tim

Reply | Threaded
Open this post in threaded view
|

Re: luaL_loadbuffer API cannot deal with shebang

Francisco Olarte
Tim:

On Thu, May 23, 2019 at 5:18 PM Tim Hill <[hidden email]> wrote:
> On May 23, 2019, at 2:25 AM, Francisco Olarte <[hidden email]> wrote:
>  if (sz>=sizeof(BOM) && memcmp(buff, p, sizeof(BOM))==0) {
>    buf+=sizeof(BOM);
>    sz+=sizeof(BOM);
>  }
>  if (sz
> Of course you mean:
> sz -= sizeof(BOM);

Yep, that's why I put the UNTESTED remark. And that's a good catch, no
compilation error ( and I doubt any compiler is smart engough to warn
on it ), and if your editor does not do illegal utf-8, and you skip
writing tests, it will probably go unnoticed for a while, and if you
have zero-padded buffers it can probably work on some files.

That's why I normally prefer [start,end) pointers. I've found you
normally only have to touch one of them and this kind of errors are
less common ( in my code ).

Francisco Olarte.