Debugging memory leaks

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Debugging memory leaks

Sean Conner

  Okay, I realize Lua has GC, so it's not a leak per se, but I am somehow
keeping references to data somewhere, and I can't seem to track them down.
The last time I encountered this issue, I was able to study the code and
through deductive reasoning, found the issue [1].

  But this time, not so much.  In general, how do others track down "lost"
references?  I just can't seem to locate it this time.

  In specific, the code (which I can't post sadly, because it's at work)
uses Lua 5.1 and is very simple.  It uses my network driver [2] which is
used in another project [3] which doesn't have the issue.  The major
difference between the two is that the "leaky" code uses TCP [4] in addition
to UDP.

  The program just accepts a packet via UDP (no response required---it's a
"fire-and-forget" type situation).  It then unpacks the data (which comes in
as CBOR), re-codes it as JSON, then then POSTs the data to an HTTP endpoint
elsewhere on the local network.  And *that's it!* That's what makes this so
maddening---it's so simple there's not obvious place for references to
accumulate, or at least, none that I can see, which is why I'm asking for
advice.

  -spc (Off to stare at code ... )

[1] "Oh, if the coroutine crashes *here,* I still have a strong
        *reference to it there* ... "

[2] https://github.com/spc476/lua-conmanorg/blob/master/lua/nfl.lua

        It's an event loop that waits for activity from a set of file
        descriptors (usually network connections) and when packets come in,
        runs a function specific to the file descriptor which can schedule a
        coroutine to run to handle any processing.

[3] Mainly, code that processes something like 60,000,000 SIP messages
        per day, but they're all delivered via UDP.

[4] https://github.com/spc476/lua-conmanorg/blob/master/lua/nfl/tcp.lua

        which uses

        https://github.com/spc476/lua-conmanorg/blob/master/lua/net/ios.lua

        to mimic the Lua file:read() API.

Reply | Threaded
Open this post in threaded view
|

Re: Debugging memory leaks

Jorge Visca
There is this from 10 years ago:
http://lua-users.org/lists/lua-l/2009-11/msg00491.html

It generates a graphviz plot that can be used to draw the data
structures in your program. I used it once and it was fun.


On 15/5/19 18:45, Sean Conner wrote:

>    Okay, I realize Lua has GC, so it's not a leak per se, but I am somehow
> keeping references to data somewhere, and I can't seem to track them down.
> The last time I encountered this issue, I was able to study the code and
> through deductive reasoning, found the issue [1].
>
>    But this time, not so much.  In general, how do others track down "lost"
> references?  I just can't seem to locate it this time.
>
>    In specific, the code (which I can't post sadly, because it's at work)
> uses Lua 5.1 and is very simple.  It uses my network driver [2] which is
> used in another project [3] which doesn't have the issue.  The major
> difference between the two is that the "leaky" code uses TCP [4] in addition
> to UDP.
>
>    The program just accepts a packet via UDP (no response required---it's a
> "fire-and-forget" type situation).  It then unpacks the data (which comes in
> as CBOR), re-codes it as JSON, then then POSTs the data to an HTTP endpoint
> elsewhere on the local network.  And *that's it!* That's what makes this so
> maddening---it's so simple there's not obvious place for references to
> accumulate, or at least, none that I can see, which is why I'm asking for
> advice.
>
>    -spc (Off to stare at code ... )
>
> [1] "Oh, if the coroutine crashes *here,* I still have a strong
> *reference to it there* ... "
>
> [2] https://github.com/spc476/lua-conmanorg/blob/master/lua/nfl.lua
>
> It's an event loop that waits for activity from a set of file
> descriptors (usually network connections) and when packets come in,
> runs a function specific to the file descriptor which can schedule a
> coroutine to run to handle any processing.
>
> [3] Mainly, code that processes something like 60,000,000 SIP messages
> per day, but they're all delivered via UDP.
>
> [4] https://github.com/spc476/lua-conmanorg/blob/master/lua/nfl/tcp.lua
>
> which uses
>
> https://github.com/spc476/lua-conmanorg/blob/master/lua/net/ios.lua
>
> to mimic the Lua file:read() API.
>

Reply | Threaded
Open this post in threaded view
|

Re: Debugging memory leaks

云风 Cloud Wu
In reply to this post by Sean Conner
I wrote a small module 

It can record all the GCobjects into another states. We can do some snapshots after an interval of time and compares the diff to find possible leaks.

Sean Conner <[hidden email]>于2019年5月16日 周四05:45写道:

  Okay, I realize Lua has GC, so it's not a leak per se, but I am somehow
keeping references to data somewhere, and I can't seem to track them down.
The last time I encountered this issue, I was able to study the code and
through deductive reasoning, found the issue [1].

  But this time, not so much.  In general, how do others track down "lost"
references?  I just can't seem to locate it this time.

  In specific, the code (which I can't post sadly, because it's at work)
uses Lua 5.1 and is very simple.  It uses my network driver [2] which is
used in another project [3] which doesn't have the issue.  The major
difference between the two is that the "leaky" code uses TCP [4] in addition
to UDP.

  The program just accepts a packet via UDP (no response required---it's a
"fire-and-forget" type situation).  It then unpacks the data (which comes in
as CBOR), re-codes it as JSON, then then POSTs the data to an HTTP endpoint
elsewhere on the local network.  And *that's it!* That's what makes this so
maddening---it's so simple there's not obvious place for references to
accumulate, or at least, none that I can see, which is why I'm asking for
advice.

  -spc (Off to stare at code ... )

[1]     "Oh, if the coroutine crashes *here,* I still have a strong
        *reference to it there* ... "

[2]     https://github.com/spc476/lua-conmanorg/blob/master/lua/nfl.lua

        It's an event loop that waits for activity from a set of file
        descriptors (usually network connections) and when packets come in,
        runs a function specific to the file descriptor which can schedule a
        coroutine to run to handle any processing.

[3]     Mainly, code that processes something like 60,000,000 SIP messages
        per day, but they're all delivered via UDP.

[4]     https://github.com/spc476/lua-conmanorg/blob/master/lua/nfl/tcp.lua

        which uses

        https://github.com/spc476/lua-conmanorg/blob/master/lua/net/ios.lua

        to mimic the Lua file:read() API.

--
Reply | Threaded
Open this post in threaded view
|

Re: Debugging memory leaks

Kim Alvefur
In reply to this post by Sean Conner
Hi,

On Wed, May 15, 2019 at 05:45:29PM -0400, Sean Conner wrote:
>   Okay, I realize Lua has GC, so it's not a leak per se, but I am somehow
> keeping references to data somewhere, and I can't seem to track them down.
> The last time I encountered this issue, I was able to study the code and
> through deductive reasoning, found the issue [1].

I did some work on a memory visualization tool that would plot used
memory over a hilbert curve. The data collection part grew into this
library: <http://luarocks.org/modules/zash/lua-mtrace>

Unfortunately I've yet to write any good tooling for finding
unexpectedly long-lived allocations, other than tediously typed out awk
programs that weren't saved.

--
Kim "Zash" Alvefur

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Debugging memory leaks

Sean Conner
In reply to this post by Sean Conner
It was thus said that the Great Sean Conner once stated:
>
>   But this time, not so much.  In general, how do others track down "lost"
> references?  I just can't seem to locate it this time.

  I want to thank Jorge, Cloud Wu, and Kim Alvefur for their responses and
links to some interesting tools and techniques.  I was able to finally track
down the reference I was keeping [1] and memory usage is now steady.

  Thinking about this, I wonder how difficult it would be to add a debug
hook for GC---call the function just prior to reclaiming the memory.  That
might help with tracking down lost references.  I mean, it kind of can be
done now with the function call hook, but you get that for every call, and
it could only really work if you have a __gc() method defined [2].  Just an
idea.

  -spc

[1] I wrote a tool last year (not released) that I can interactively
        check a Lua state (using a text-based UI).  I was able to integrate
        it into the event framework, so I was able to check various tables
        in real time (as the program was processing network events) to see
        what table was growing without bound.

        Attached is a screen shot for those that might be interested.

[2] I think---I haven't actually tested that theory yet.

Reply | Threaded
Open this post in threaded view
|

Re: Debugging memory leaks

Sean Conner
It was thus said that the Great Sean Conner once stated:
>
> [1] I wrote a tool last year (not released) that I can interactively
> check a Lua state (using a text-based UI).  I was able to integrate
> it into the event framework, so I was able to check various tables
> in real time (as the program was processing network events) to see
> what table was growing without bound.
>
> Attached is a screen shot for those that might be interested.

  I forgot the screen shot.  Sigh.

  -spc (Included this time ...)

luaview.gif (9K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Debugging memory leaks

nobody
On 16/05/2019 20.30, Sean Conner wrote:

> It was thus said that the Great Sean Conner once stated:
>>
>> [1] I wrote a tool last year (not released) that I can
>> interactively check a Lua state (using a text-based UI).  I was
>> able to integrate it into the event framework, so I was able to
>> check various tables in real time (as the program was processing
>> network events) to see what table was growing without bound.
>>
>> Attached is a screen shot for those that might be interested.
>
> I forgot the screen shot.  Sigh.
>
> -spc (Included this time ...)

That sounds cool!  Can you briefly describe how you get the information
from the Lua state to the tool?  (Apart from invasive hooks everywhere &
blasting stuff out via sockets, I'd have no idea how to do this…)

-- nobody

Reply | Threaded
Open this post in threaded view
|

Re: Debugging memory leaks

Sean Conner
It was thus said that the Great nobody once stated:

> On 16/05/2019 20.30, Sean Conner wrote:
> >It was thus said that the Great Sean Conner once stated:
> >>
> >>[1] I wrote a tool last year (not released) that I can
> >>interactively check a Lua state (using a text-based UI).  I was
> >>able to integrate it into the event framework, so I was able to
> >>check various tables in real time (as the program was processing
> >>network events) to see what table was growing without bound.
> >>
> >>Attached is a screen shot for those that might be interested.
> >
> >I forgot the screen shot.  Sigh.
> >
> >-spc (Included this time ...)
>
> That sounds cool!  Can you briefly describe how you get the information
> from the Lua state to the tool?  (Apart from invasive hooks everywhere &
> blasting stuff out via sockets, I'd have no idea how to do this…)

  No invasive hooks, and use of the standard debug module.

  It's interactive at the command line and I typically use it from the stock
Lua interpeter:

[spc]lucy:~>lua
Lua 5.3.5  Copyright (C) 1994-2018 Lua.org, PUC-Rio
> luaview = require "org.flummux.luaview"
> luaview.run()

  But because it's a module and *not* an application, you can embed it.  So
I added it to the code in question:

        -- This is UDP socket that accepts incoming requests.  It will
        -- then spawn a coroutine to handle sending the data to an HTTP
        -- endpoint

        nfl.SOCKETS:insert(usock,'r',function() -- [1]
          local remote,packet,err = usock:recv()
          nfl.spawn(handler,remote,packet)
        end)

        luaview = require "org.flummux.luaview"
        luaview.init() -- intialize the luaview module

        -- When a keyboard event is detected, run one step
        -- of the luaview module.  This give us a real-time
        -- interactive view into the current Lua state as the
        -- program is running.  I can check the global environment,
        -- the registry, upvalues, metatables, etc.

        nfl.SOCKETS:insert(luaview,'r',function()
          if not luaview.step() then
            nfl.SOCKETS;remove(luaview)
            luaview.close()
          end
        end)

        nfl.server_eventloop()

  So it's not a remote, interactive viewer thing---it's purely a local,
interactive thing.

  The function luaview.step() just reads the next key from the keyboard and
handles that one key.  luaview.run() calls luaview.init(), luaview.step()
(in a loop) and luaview.close().  That's pretty much it to the luaview API.

  -spc (Never given it thought of making it remote ... )

[1] My wrapper for select() [2] will attempt to call _tofd() via a
        metatable on the second parameter.  My other modules, like
        org.conman.net (and org.flummux.luaview) support this.  If you use
        org.conman.fsys, it will add a _tofd() function to the FILE*
        metatable in Lua so one can pass in items like io.stdin [3].

[2] Which will use epoll() under Linux, poll() under other Unix systems,
        and only if poll() isn't available, select().  The API is the same
        across all three implementations.

[3] I'm still unsure if monkey patching like this is a Good Thing or
        not.

Jim
Reply | Threaded
Open this post in threaded view
|

Re: Debugging memory leaks

Jim
In reply to this post by Sean Conner
On 5/16/19, Sean Conner <[hidden email]> wrote:
>> Attached is a screen shot for those that might be interested.
> I forgot the screen shot.  Sigh.

do you still use fvwm these days ? :D

Reply | Threaded
Open this post in threaded view
|

Re: Debugging memory leaks

Sean Conner
It was thus said that the Great Jim once stated:
> On 5/16/19, Sean Conner <[hidden email]> wrote:
> >> Attached is a screen shot for those that might be interested.
> > I forgot the screen shot.  Sigh.
>
> do you still use fvwm these days ? :D

  Yup.

  -spc


Reply | Threaded
Open this post in threaded view
|

Re: Debugging memory leaks

Jim-2
17.05.2019, 15:28, "Sean Conner" <[hidden email]>:
> It was thus said that the Great Jim once stated:
>>  On 5/16/19, Sean Conner <[hidden email]> wrote:
>>  >> Attached is a screen shot for those that might be interested.
>>  > I forgot the screen shot. Sigh.
>>
>>  do you still use fvwm these days ? :D
>
>   Yup.

is that killer .fvwm(2 ?? ;-)rc available online ?
i am interesting in finding out how you set up those
kickass buttons you proudly presented to us in this
screenshot ! :D

wasn't this screenshot in fact presented here to show us
those nice buttons you use since the 90ies ? :PP

BTW:

this is not totally off topic:
has anyone written a fvwm Lua module ("FvwmLua" :-)
resembling the provided "FvwmPerl" ?

though i would place the Lua interpreter directly into such a
module (i. e. link liblua.a into it), unlike "FvwmPerl" did
(this is just a perl script that loads further perl code from a
conventional perl module as far as i can recall) ...


Jim
Reply | Threaded
Open this post in threaded view
|

Re: Debugging memory leaks

Jim
On 5/17/19, Jeff <[hidden email]> wrote:
> has anyone written a fvwm Lua module ("FvwmLua" :-)
> resembling the provided "FvwmPerl" ?

i wonder if  there are any window managers other than
"awesome" that use Lua ?

i am more interested in those that unlike awesome
(which embeds Lua) are written entirely as Lua script
that uses C extensions to interact with X11 akin to
"subtle" vs Ruby and "whim" vs Tcl/Tk.

does anyone know any such (preferably tiling) WMs ?

Reply | Threaded
Open this post in threaded view
|

Re: Debugging memory leaks

dyngeccetor8
On 18/05/2019 21.45, Jim wrote:

> On 5/17/19, Jeff <[hidden email]> wrote:
> i wonder if  there are any window managers other than
> "awesome" that use Lua ?
>
> i am more interested in those that unlike awesome
> (which embeds Lua) are written entirely as Lua script
> that uses C extensions to interact with X11 akin to
> "subtle" vs Ruby and "whim" vs Tcl/Tk.
>
> does anyone know any such (preferably tiling) WMs ?
>
TekUI http://tekui.neoscientists.org/

I like it and use in some of my projects.

-- Martin

Reply | Threaded
Open this post in threaded view
|

Re: Debugging memory leaks

Lorenzo Donati-3
On 18/05/2019 23:43, dyngeccetor8 wrote:

> On 18/05/2019 21.45, Jim wrote:
>> On 5/17/19, Jeff <[hidden email]> wrote:
>> i wonder if  there are any window managers other than
>> "awesome" that use Lua ?
>>
>> i am more interested in those that unlike awesome
>> (which embeds Lua) are written entirely as Lua script
>> that uses C extensions to interact with X11 akin to
>> "subtle" vs Ruby and "whim" vs Tcl/Tk.
>>
>> does anyone know any such (preferably tiling) WMs ?
>>
> TekUI http://tekui.neoscientists.org/
>
> I like it and use in some of my projects.
>
> -- Martin
>
>

That seems nice and lean. I really would need a GUI framework that
allows getting a simple GUI up and running in minutes.

I use mainly Windows and often I have longed for a GUI manager that
allowed me to make some quick and dirty scripts more interactive. I
usually did it in wxLua, but it was a bit overkill for very small
scripts (<500 LoC), moreover it seems to be dead now.

Is TekUI still actively developed?

The last release is v. 1.12 which dates back to 2015. I see in the
download directory a development snapshot committed this week, so it
seems that it is not abandonware but no stable release in years is suspect.

It happened to wxLua as well (on which I invested some time), then the
project apparently died, so I'm a bit wary to pick another up when there
is no "sign of life" from the devs.

Cheers!

-- Lorenzo


Reply | Threaded
Open this post in threaded view
|

Re: Debugging memory leaks

Victor Bombi
I keep using wxLua. There is also https://webserver2.tecgraf.puc-rio.br/iup/
Or for a superminimal (But from LuaJIT) I made https://github.com/sonoro1234/LuaJIT-ImGui

> El 19 de mayo de 2019 a las 10:23 Lorenzo Donati <[hidden email]> escribió:
>
>
> On 18/05/2019 23:43, dyngeccetor8 wrote:
> > On 18/05/2019 21.45, Jim wrote:
> >> On 5/17/19, Jeff <[hidden email]> wrote:
> >> i wonder if  there are any window managers other than
> >> "awesome" that use Lua ?
> >>
> >> i am more interested in those that unlike awesome
> >> (which embeds Lua) are written entirely as Lua script
> >> that uses C extensions to interact with X11 akin to
> >> "subtle" vs Ruby and "whim" vs Tcl/Tk.
> >>
> >> does anyone know any such (preferably tiling) WMs ?
> >>
> > TekUI http://tekui.neoscientists.org/
> >
> > I like it and use in some of my projects.
> >
> > -- Martin
> >
> >
>
> That seems nice and lean. I really would need a GUI framework that
> allows getting a simple GUI up and running in minutes.
>
> I use mainly Windows and often I have longed for a GUI manager that
> allowed me to make some quick and dirty scripts more interactive. I
> usually did it in wxLua, but it was a bit overkill for very small
> scripts (<500 LoC), moreover it seems to be dead now.
>
> Is TekUI still actively developed?
>
> The last release is v. 1.12 which dates back to 2015. I see in the
> download directory a development snapshot committed this week, so it
> seems that it is not abandonware but no stable release in years is suspect.
>
> It happened to wxLua as well (on which I invested some time), then the
> project apparently died, so I'm a bit wary to pick another up when there
> is no "sign of life" from the devs.
>
> Cheers!
>
> -- Lorenzo
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Debugging memory leaks

szbnwer@gmail.com
hi all! :)

(@spectators: sorry for continuing the off-topic, im using the dummy
variant of gmail, thats hard to be fine-tuned...)

@Lorenzo:
Timm S. Müller (one of the developers of tekui, probably the main dev)
is responsive, and he just wrote in these days something unrelated to
lua-l, so u could ask him about whatever.

btw ive talked a bit with him, maybe like a yr ago, on their mailing
list, so i have some answers right from him, but sorry if anything is
outdated or incorrect:
so he have said, that he accumulated some minor stuffs by the time
that he still didnt have time to merge; he said to me, that i can fork
it if i want, so he wouldnt mind/be happy with that; and that tekui is
kinda feature complete.

actually it really serves very well, and its easy to play with it, if
u wanna make anything that is not already solved, and also, i dunno
any better tool out there for gui+graphics stuffs. iup is using
bindings for its cooler abilities, as well as for its basics, so ive
found it heavy, like i dont really wanna mess with those in case of
need to dig down, but its more popular cuz its from puc-rio and cuz
its native windows, and dunno what else.

(here, ull see some nasty stuffs from a developer's perspective about
me, but reasoning will come after.) so actually there are some really
minor glitches around, that wont harm nor seem to be hard to fix, and
tekui is otherwise reliable, (i started to use it in the beginning of
2015).
- the dropdowns of the window menu are misplaced initially (at least
with the x11 driver, as ive never used the others), but after moving
the window, they are fine, so they have the necessary math, but the
initialization dont get it, and ive never even tried to fix it, just
cuz i didnt really care enough to give it the priority.
- an another minor glitch is that in multiline inputs wont drop the
last char into the next line if u hit the enter right before them, so
in those cases i just put an extra char (space) there, break the line,
and then remove the extra char, while that would be really an easy to
fix stuff, as only 1 long function should be fixed for it that handles
the keypresses, just i again didnt care enough to fix it.
- an another multiline input related stuff is when they get the focus
(so not in the meantime of editing whatever text), then they
autoscroll like the prompt goes to the 1st line, and when u wanna
select some text combined with this, then the selection background
seems weird, but otherwise works even then, i think disabling the
reaction on getting focus would resolve both and can be really easy,
but again, i didnt care enough.
- the worse bug ive found by the years is that "őűŐŰ" chars get the
wrong letters on keypress (but not on copypaste), so ive monkeypatched
that with a simple replacement hash table, cuz that was a
hard-to-live-with issue.
- a hard one, that ive tried to resolve, but didnt reach the point so
far now, is that the clipboard works in tekui, even between
independent tekui apps, as well as inserting anything from sys-wide
clipboard into tekui, but not vice-versa (at least on linux, but i
just wont use win or mac), so i can only get texts out from it via
files or terminal output, but actually i could live with it so far
now.

the promised reasoning for not fixing up the found issues, is that i
dont wanna make a legacy code base from it before i make some tools to
track them, and also to see if they wont mess up anything elsewhere,
but later ive found out that i can just insert './?.lua;' to the
beginning of package.path, and make a tek dir into my base dir, and
there i can have modified files without making too much mess, but that
came into my mind after ive met with most of those issues, so i just
didnt have the arrival for fixing them (also, the monkey patch as
well! :D ) nor they got the priority to doing so, as im about to make
some tools to make those tasks smoother from various perspectives...

after all, i still wouldnt switch to anything else, even if i keep an
eye on the rest of the world near to and far from lua as well. its
really fast, flexible (even in runtime), small (one can really do
whatever with it as a whole, as it can be read from the 1st byte to
the last one), not relying on huge uncontrollable stuffs (or better
said, only relies on x11 or win graphics stuffs according to those
drivers in it, but the graphics driver can be changed, and others are
available for it), its really easy to read, and easy to play with it,
and it have all the essentials to do anything with it.

on the contrary, if u dont wanna hack weird graphical elements like i
plan to do so, but only want to have basics like input, checkbox,
listers and whatever with some layouting and styling, then maybe u can
find something that is more suitable for ur actual needs.

first, when i started to play with tekui, i was puzzled about a bunch
of things in it, but currently ive got a kinda good understanding
about it, and i already dug down into most of the deepest depths of
the source, so if u think u could give it a shot, then feel free to
ask me, or write to the mailing list of tekui, where ill find u, and i
can help u in whatever, and i can do it happily, as i really adore
tekui. :) otherwise i didnt make a fork, cuz i dont care about win and
mac, and i also wanna destroy its whole oop, as i have a different
unique organization technique, and i also wanna make much
customization around it, but probably that wont happen soon... so if i
would make a public fork, then others maybe wouldnt be happy with my
modifications if they wanna rely on it as it is now. otherwise i didnt
really throw away the idea, of doing so, so if i find serious company,
then i can even follow its current guidelines and what not... :D

so yepp, feel free to ask whatever u want, i think/hope i can help in
much, even if my free time is kinda chaotic. :)

bests! :)