Lua 5.4.2 (realloc ?)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Lua 5.4.2 (realloc ?)

Ranier Vilela-2
Hi,

With some of my tests, I came across this alert from DrMemory (https://drmemory.org/)
~~Dr.M~~ Error #7: POSSIBLE LEAK 1616 direct bytes 0x0000028e94a6b8d0-0x0000028e94a6bf20 + 7442 indirect bytes
~~Dr.M~~ # 0 replace_realloc               [d:\drmemory_package\common\alloc_replace.c:2669]
~~Dr.M~~ # 1 lua5.4.2.dll!l_alloc          [C:\dll\Lua\lauxlib.c:995]
~~Dr.M~~ # 2 lua5.4.2.dll!luaL_newstate    [C:\dll\Lua\lauxlib.c:1038]
~~Dr.M~~ # 3 main                          [C:\dll\Lua\lua.c:625]

I'm surprised that a simple test with very little used memory,
Lua is already calling realloc!

regards,
Ranier Vilela
Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.4.2 (realloc ?)

Luiz Henrique de Figueiredo
> With some of my tests, I came across this alert from DrMemory (https://drmemory.org/)
> ~~Dr.M~~ Error #7: POSSIBLE LEAK 1616 direct bytes 0x0000028e94a6b8d0-0x0000028e94a6bf20 + 7442 indirect bytes
> ~~Dr.M~~ # 0 replace_realloc               [d:\drmemory_package\common\alloc_replace.c:2669]
> ~~Dr.M~~ # 1 lua5.4.2.dll!l_alloc          [C:\dll\Lua\lauxlib.c:995]
> ~~Dr.M~~ # 2 lua5.4.2.dll!luaL_newstate    [C:\dll\Lua\lauxlib.c:1038]
> ~~Dr.M~~ # 3 main                          [C:\dll\Lua\lua.c:625]
>
> I'm surprised that a simple test with very little used memory,
> Lua is already calling realloc!

No surprises here: luaL_newstate calls lua_newstate which allocates
memory for the Lua state using realloc as malloc.
Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.4.2 (realloc ?)

Francisco Olarte
In reply to this post by Ranier Vilela-2
Rainier:

On Mon, Nov 9, 2020 at 2:55 PM Ranier Vilela <[hidden email]> wrote:
> I'm surprised that a simple test with very little used memory,
> Lua is already calling realloc!

Why ( are you surprised ) ? ( It may be, and given what I have read in
the manual I suspect is) using realloc for all memory allocations.
AAMOF a quick grep in the sources show no malloc usages and just a
single realloc and free ones ( you can do everything with realloc ).

Francisco Olarte.
Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.4.2 (realloc ?)

Ranier Vilela-2
Em seg., 9 de nov. de 2020 às 13:04, Francisco Olarte <[hidden email]> escreveu:
Rainier:

On Mon, Nov 9, 2020 at 2:55 PM Ranier Vilela <[hidden email]> wrote:
> I'm surprised that a simple test with very little used memory,
> Lua is already calling realloc!

Why ( are you surprised ) ? ( It may be, and given what I have read in
the manual I suspect is) using realloc for all memory allocations.
AAMOF a quick grep in the sources show no malloc usages and just a
single realloc and free ones ( you can do everything with realloc ).
Including lost the original pointer, if realloc fail.

regards,
Ranier Vilela
Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.4.2 (realloc ?)

Roberto Ierusalimschy
In reply to this post by Francisco Olarte
> > I'm surprised that a simple test with very little used memory,
> > Lua is already calling realloc!
>
> Why ( are you surprised ) ? ( It may be, and given what I have read in
> the manual I suspect is) using realloc for all memory allocations.
> AAMOF a quick grep in the sources show no malloc usages and just a
> single realloc and free ones ( you can do everything with realloc ).

Indeed, Lua uses realloc for all its allocations. According to ISO C99:

  7.20.3.4 The realloc function

  Synopsis
  #include <stdlib.h>
  void *realloc(void *ptr, size_t size);

  [...]
  If ptr is a null pointer, the realloc function behaves like the malloc
  function for the specified size.

-- Roberto
Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.4.2 (realloc ?)

Luiz Henrique de Figueiredo
In reply to this post by Ranier Vilela-2
> Including lost the original pointer, if realloc fail.

realloc leaves the original memory and poiner intact if it fails. Lua
is very careful about this point.
Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.4.2 (realloc ?)

Francisco Olarte
In reply to this post by Ranier Vilela-2
Rainier:

On Mon, Nov 9, 2020 at 5:20 PM Ranier Vilela <[hidden email]> wrote:
> Em seg., 9 de nov. de 2020 às 13:04, Francisco Olarte <[hidden email]> escreveu:
...
>> single realloc and free ones ( you can do everything with realloc ).
> Including lost the original pointer, if realloc fail.

Only if you are not careful and do "return realloc(old, newsize)" or
something like that. You can equally easily lose the old pointer if
you try malloc or any other function and do not program it right. This
is C memory management 101.

Francisco Olarte.
Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.4.2 (realloc ?)

Ranier Vilela-2
Em seg., 9 de nov. de 2020 às 13:54, Francisco Olarte <[hidden email]> escreveu:
Rainier:

On Mon, Nov 9, 2020 at 5:20 PM Ranier Vilela <[hidden email]> wrote:
> Em seg., 9 de nov. de 2020 às 13:04, Francisco Olarte <[hidden email]> escreveu:
...
>> single realloc and free ones ( you can do everything with realloc ).
> Including lost the original pointer, if realloc fail.

Only if you are not careful and do "return realloc(old, newsize)" or
something like that. You can equally easily lose the old pointer if
you try malloc or any other function and do not program it right. This
is C memory management 101.
Like this:
l_alloc function (lauxlib.c)
995: return realloc(ptr, nsize);

regards,
Ranier Vilela
Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.4.2 (realloc ?)

Francisco Olarte
Rainier:

On Mon, Nov 9, 2020 at 6:07 PM Ranier Vilela <[hidden email]> wrote:
> Em seg., 9 de nov. de 2020 às 13:54, Francisco Olarte <[hidden email]> escreveu:
...
>> you try malloc or any other function and do not program it right. This
>> is C memory management 101.
>
> Like this:
> l_alloc function (lauxlib.c)
> 995: return realloc(ptr, nsize);

Did you bother reading the docs? Specifically the phrase "When nsize
is not zero, the allocator must behave like realloc. " ( btw, that
usage is, IIRC, protected inside the else branch of a nice "if
(nsize==0)".

If you want to look for ptr leaks you have to search for the usages of
the l_alloc function passed and stored inside the state. I believe
they are confined inside lmem.* but i'm not sure of that one.

Francisco Olarte.
Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.4.2 (realloc ?)

Luiz Henrique de Figueiredo
In reply to this post by Ranier Vilela-2
> l_alloc function (lauxlib.c)
> 995: return realloc(ptr, nsize);

Check https://www.lua.org/source/5.3/lauxlib.c.html#l_alloc
I think this will be in Lua 5.4.2 too.
Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.4.2 (realloc ?)

Ranier Vilela-2
In reply to this post by Francisco Olarte
Em seg., 9 de nov. de 2020 às 14:25, Francisco Olarte <[hidden email]> escreveu:
Rainier:

On Mon, Nov 9, 2020 at 6:07 PM Ranier Vilela <[hidden email]> wrote:
> Em seg., 9 de nov. de 2020 às 13:54, Francisco Olarte <[hidden email]> escreveu:
...
>> you try malloc or any other function and do not program it right. This
>> is C memory management 101.
>
> Like this:
> l_alloc function (lauxlib.c)
> 995: return realloc(ptr, nsize);

Did you bother reading the docs? Specifically the phrase "When nsize
is not zero, the allocator must behave like realloc. " ( btw, that
usage is, IIRC, protected inside the else branch of a nice "if
(nsize==0)".

If you want to look for ptr leaks you have to search for the usages of
the l_alloc function passed and stored inside the state. I believe
they are confined inside lmem.* but i'm not sure of that one.
Realloc only is secure like this:
  (void)ud; (void)osize;  /* not used */
  if (nsize == 0) {
    free(ptr);
    return NULL;
  }
  else
  {
void * newptr;
newptr = realloc(ptr, nsize);
if (newptr == NULL)
return ptr;
return newptr;
  }

Ranier Vilela
Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.4.2 (realloc ?)

Francisco Olarte
Rainier:

On Mon, Nov 9, 2020 at 6:29 PM Ranier Vilela <[hidden email]> wrote:

> Realloc only is secure like this:
>   (void)ud; (void)osize;  /* not used */
>   if (nsize == 0) {
>     free(ptr);
>     return NULL;
>   }
>   else
>   {
> void * newptr;
> newptr = realloc(ptr, nsize);
> if (newptr == NULL)
> return ptr;
> return newptr;
>   }

That is plain wrong for l_alloc function. Look at the docs. It says
you must behave like realloc, and you are not doing it. These is the
least of your problems.

In your implementation you are returning a valid pointer when realloc
fails. So if I call your function to grow a block, and it can not grow
it I will receive a valid ptr, then use it, write out of boundaries
and chaos ensues. And I've used several implementations of realloc on
which it can grow the block without copying it ( typical cases are
when allocator internally rounds up block for certain size ranges and
when you use something similar to sbrk and grow the top allocated
block ).

In fact this function of yours is plain wrong. If you assume you can
detect failure by testing the returned pointer ( which is not allways
true ) you still have to keep a copy outside for that test, so just
use realloc.

OTOH, the bundled l_alloc which behaves like realloc is perfectly
safe, as plain realloc is. Everybody (which wants to recover from
allocation failures, the ones not wanting to bother use abort on error
wrapper, like the classic xmalloc/xrealloc many libs provide ) knows
than when you call realloc you have to keep a copy of the ptr if you
do not want to loose track on allocation failures. l_alloc does NOT
want to recover from allocation failures, it's just a convenience
wrapper for realloc. The callers of the l_alloc do, and are the ones
any knowledgeable C programmer would look at to hunt for memory leaks,
just from the description of the interface in the manual.

Francisco Olarte.
Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.4.2 (realloc ?)

Luiz Henrique de Figueiredo
In reply to this post by Luiz Henrique de Figueiredo
> Check https://www.lua.org/source/5.3/lauxlib.c.html#l_alloc
> I think this will be in Lua 5.4.2 too.

Actually, Lua 5.4.2 handles l_alloc failures and so l_alloc in 5.4 is
back to its simpler version. Sorry for the noise.