lua hacking wondering

classic Classic list List threaded Threaded
35 messages Options
12
Reply | Threaded
Open this post in threaded view
|

lua hacking wondering

Jeffrey Drake
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I am trying to install an '__existingindex' metatable method. But not quite working.

The code I am trying to do: (excuse the lack of a patch, but I am using a custom directory arrangement for my lua framework (osx))

ltm.c @ 28
void luaT_init (lua_State *L) {
  static const char *const luaT_eventname[] = {  /* ORDER TM */
    "__index", "__existingindex", "__newindex", "__usedindex",

ltm.h @ 18
typedef enum {
  TM_INDEX,
  TM_EXISTINGINDEX,
  TM_NEWINDEX,

lvm.c @ 147

/*
** Function to index a table.
** Receives the table at `t' and the key at `key'.
** leaves the result at `res'.
*/
const TObject *luaV_gettable (lua_State *L, const TObject *t, TObject *key,
                              int loop) {
  if (loop > MAXTAGLOOP)
    luaG_runerror(L, "loop in gettable");
  if (ttistable(t)) {  /* `t' is a table? */
    Table *h = hvalue(t);

    /* start EXISTINGINDEX HACK  */
    const TObject *tm = fasttm(L, h->metatable, TM_EXISTINGINDEX);
    if (tm != NULL)
    {
        if (ttisfunction(tm))
        {
            callTMres(L, tm, t, key);
            return L->top;
        }
    }

    /* end HACK */

    const TObject *v = luaH_get(h, key);  /* do a primitive get */
    if (!ttisnil(v)) return v;
    else return luaV_index(L, t, key, loop+1);
  }
  else return luaV_getnotable(L, t, key, loop+1);
}


I do not claim to understand lua internals very well, but I basically could figure out all of it but how the loops were working.

This is how it works:

Anubis:~/.tmp/lua-5.0.2/src/lua jdrake$ rlwrap lua
Lua 5.0.2  Copyright (C) 1994-2004 Tecgraf, PUC-Rio
> blah = {}
> blah.x = 5
> print (blah.x)
5
> ximp = {}
> function ximp.__existingindex(t,k) print("existing index: ", k); return k end
> setmetatable(blah, ximp)
> print (blah.x)
5
>

>From what I can tell the function definitely isn't being called.

An example of __index being called:

> function ximp.__index(t,k) print("index: ", k); return 0 end
> print (blah.y)
index:  y
0


Any ideas here?


"Reportedly, he remarked to one of his Christian missionary friends: 'What you teach us to do is admirable, but what you teach us to believe is foolish'."
- - Referring to King Mongkut of Siam
http://en.wikipedia.org/wiki/Mongkut
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (Darwin)

iD8DBQFA0Ryo0noMGorKkKgRAszoAJ4qmpLslNKOG4A4UXmNwpdBMeI7vACfafPQ
ZdP6RG3hGY/dHJTrINYp+Z8=
=IKYA
-----END PGP SIGNATURE-----


Reply | Threaded
Open this post in threaded view
|

Re: lua hacking wondering

Diego Nehab-3
Hi,

> I am trying to install an '__existingindex' metatable method. But not
> quite working.

Can't you get away with using a proxy table? That way you always get
the __index or __index in the proxy table and can refer to the real
table to check if the key is already there.

[]s,
Diego.

Reply | Threaded
Open this post in threaded view
|

Re: lua hacking wondering

Jeffrey Drake
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

that is the backup plan, but I also want to be able to figure out why it isn't working.

I have figured the problem a little better:

A modification:
    const TObject *tm = luaT_gettmbyobj(L, t, TM_EXISTINGINDEX);

    //fasttm(L, h->metatable, TM_EXISTINGINDEX);
    printf("i1\n");
    if (!ttisnil(tm))
        printf("h2\n");

        if (ttisfunction(tm))
        {
            printf("h3\n");
            callTMres(L, tm, t, key);
            return L->top;
        }
   // }

    /* end HACK */

1 is always printed; 2, 3 are not. So the problem I figure has to be that it is nil. Which simply should not be from my current understanding.


On Jun 17, 2004, at 01:48, [hidden email] wrote:

Hi,

I am trying to install an '__existingindex' metatable method. But not
quite working.

Can't you get away with using a proxy table? That way you always get
the __index or __index in the proxy table and can refer to the real
table to check if the key is already there.

[]s,
Diego.

"Reportedly, he remarked to one of his Christian missionary friends: 'What you teach us to do is admirable, but what you teach us to believe is foolish'."
- - Referring to King Mongkut of Siam
http://en.wikipedia.org/wiki/Mongkut
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (Darwin)

iD8DBQFA0Tby0noMGorKkKgRAiLxAJ9bat7jLGRl33/7q6YEOOw/28SoEwCcDxl8
vASsp9M9/EyDVrh8J02WGyo=
=JaGp
-----END PGP SIGNATURE-----


Reply | Threaded
Open this post in threaded view
|

Re: lua hacking wondering

Diego Nehab-3
In reply to this post by Diego Nehab-3
Hi,

I hate reading my messages again...

> Can't you get away with using a proxy table? That way you always get
> the __index or __index in the proxy table and can refer to the real
                   ^^^^^
       I meant "__newindex", of course

> table to check if the key is already there.

[]s,
Diego.

Reply | Threaded
Open this post in threaded view
|

Re: lua hacking wondering

Mark Hamburg-4
I modified Lua to support a __write metamethod which if present gets called
whether the index exists or not. The semantics relative to __newindex get
tricky to define but not because of the code. The code itself was quite
straightforward as I recall and made me feel good about how well thought out
the base Lua implementation was.

Now comes a report of an effort to create an __existingindex metamethod.
(From my experience, I would say start by tracing through the handling of
the existing __index metamethod.)

Always, the standard answer is to use proxy tables. These work, but are
somewhat expensive with respect to space and they break in other ways such
as with respect to table iteration. They also won't catch manipulation
through routines like table.insert. They are also clumsy to set up if the
table already exists and is populated.

Here is a quick summary of ideas (some mine, some from others) for improving
these fairly common design patterns for Lua. They have some overlap and it
might not be wise to do all of them. I post them here to encourage comments.

* Add table.swap( a, b ) which swaps the contents of two tables a and b.
This is useful when converting existing tables into proxies since it keeps
all of the old references working.

* Add a protected mode to tables that would force use of __index and
__newindex. (It might be nice to have the option to just do
write-protection.)

* Add metamethods for the other standard table access (e.g., iteration) and
manipulation (table namespace) routines.

* Give tables a default metatable and shift from writing:

    table.insert( t, v )

To writing:

    t:insert( v )

This can mostly be done now by setting table as the metatable for an empty
table, but it isn't done automatically and hence code is more likely to use
the first form. The second form, however, makes it easier to override the
implementations.

Mark


Reply | Threaded
Open this post in threaded view
|

RE: lua hacking wondering

Vijay Aswadhati-2
<begin rambling>
I have seen "use the proxy table" answers on one too many occasions. 

Before the flame war hungry folks jump in, let me state up front that
Lua is a great language, simple, extensible, what have you and I will
not blink my eyes should you throw some miracle at me that is possible
with Lua.

I am not sure about others (but I suspect there are a few who share my
sentiment), but in -=systems integrations=- (where scripting languages are
playing a crucial role more and more) having to UNDERSTAND and implement
the "proxy table design pattern" is a bummer [for the "menial" programmers]
because it stands in the way of a "speedy" solution - no matter how
trivial the implementation MAY be. This could easily suck away anywhere
from 2 - 8 hours of searching, having the "hmmm....aha" moment and finally
the implementation game.

The fact that this question has come up so many times would suggest that
there is a cause for reifying the design pattern into a primitive (i.e. 
integrated into the core) - unless this causes excessive code bloat,
significant performance loss or the core becomes non portable.

It's not that I don't want to understand this pattern; I just do not
have the time and I do not want to create it for every project that
I work with that uses Lua.

</end rambling>

So the question is this: is it worth requesting this as a feature in
5.1?

cheers
Vijay Aswadhati



Reply | Threaded
Open this post in threaded view
|

RE: lua hacking wondering

Virgil Smith
> <begin rambling>
> I have seen "use the proxy table" answers on 
> one too many occasions.
...
> </end rambling>

> So the question is this: is it worth requesting this 
> as a feature in 5.1?

>From a confused observer:
Exactly what are you asking be included?



Reply | Threaded
Open this post in threaded view
|

Re: lua hacking wondering

Asko Kauppi-3
In reply to this post by Vijay Aswadhati-2

I agree with your points and, personally, try to avoid metatables and whatever "sophisticated Lua" whenever possible. Just to keep things simple.

In fact, I would regard metatables, proxys, weak tables.. as an "advanced Lua" altogether, which the normal programmer, or non-programmer even less :) does not need to master.

The situation is quite the same as with C++, providing -on the one hand- mysterious ways of making classes, templates, blahblah and on the other - the nice ease of use if those are done right.

For any organization using Lua, I would regard 1-2 persons knowing it "insight out" to be enough. The number of people using it for basic programming (and being _very_ productive in it!) can be hundreds.

-ak



21.6.2004 kello 02:39, Vijay Aswadhati kirjoitti:

 <begin rambling>
I have seen "use the proxy table" answers on one too many occasions.

Before the flame war hungry folks jump in, let me state up front that
Lua is a great language, simple, extensible, what have you and I will
not blink my eyes should you throw some miracle at me that is possible
with Lua.

I am not sure about others (but I suspect there are a few who share my
sentiment), but in -=systems integrations=- (where scripting languages are playing a crucial role more and more) having to UNDERSTAND and implement the "proxy table design pattern" is a bummer [for the "menial" programmers]
because it stands in the way of a "speedy" solution - no matter how
trivial the implementation MAY be. This could easily suck away anywhere
from 2 - 8 hours of searching, having the "hmmm....aha" moment and finally
the implementation game.

The fact that this question has come up so many times would suggest that
there is a cause for reifying the design pattern into a primitive (i.e.
integrated into the core) - unless this causes excessive code bloat,
significant performance loss or the core becomes non portable.

It's not that I don't want to understand this pattern; I just do not
have the time and I do not want to create it for every project that
I work with that uses Lua.

</end rambling>

So the question is this: is it worth requesting this as a feature in
5.1?

cheers
Vijay Aswadhati




Reply | Threaded
Open this post in threaded view
|

Re: lua hacking wondering

benjamin sunshine-hill
In reply to this post by Virgil Smith
On Sunday 20 June 2004 05:20 pm, Virgil Smith wrote:
> >From a confused observer:
>
> Exactly what are you asking be included?

I assume he's requesting support for a __setindex metamethod, that would be 
called whenever a value in a table is set. The number one usage of proxy 
tables is to force __newindex to be called each time a table value is set, 
rather than only when a new one is set.

Reply | Threaded
Open this post in threaded view
|

RE: lua hacking wondering

Vijay Aswadhati-2
In reply to this post by Virgil Smith
>> So the question is this: is it worth requesting this
>> as a feature in 5.1?

>>From a confused observer:
>Exactly what are you asking be included?

One of those occasions when following the netiquette (of snipping...) leads
to
confused observers!

To answer your question, the posting was in response to Jeffrey's
posting on his attempts to "'__existingindex' metatable method"
and the patch he provided for discussion which was followed up by
Diego's well meaning "Can't you get away with using a proxy table..."
response.

This was followed by Mark (Hamburg) comments on how he modified Lua to
support a __write metamethod and his remarks about the "standard answer
is to use proxy tables" and some suggestions he had.

My posting was prompted by the above discussion and contemplation
on why what seems to be a common use case (from the many postings
on this news group) can't be provided by the core instead of
everyone (in need) having to patch it.

Which leads nicely to the discussion of the standard philosophy implicit
in this mailing list of how one or two experts in Lua should handle all
the gore and others should just use what these experts decide for them.

I personally have no problems with that line of thought. But it
falls short for systems designers (in small development houses) who are
not experts in a language, whose task is to focus on the "SYSTEM" of which
a scripting language is a subsystem; and would like to carve solutions from
existing components with the least effort and minimal discovery.

While I was writing this up, Ben Sunshine-Hill suggested that I am
perhaps looking for __setindex metamethod; yes, perhaps.

My rambling was more to make use of Jeffrey's post as an opportunity
to bring to the community's attention of a recurring use case - the
use of proxy table, how curiously recurring pattern it seems to be.

And to hear from the community:
- why some developers are going down the path of modifying the core?
- are they right in their approach?
- is it worth reifying this "curiously recurring pattern" as feature
  of 5.1 so that others do not have to touch the core.

Hope that clears up the confusion.

Vijay Aswadhati



Reply | Threaded
Open this post in threaded view
|

Re: lua hacking wondering

Philippe Lhoste
In reply to this post by Vijay Aswadhati-2
Vijay Aswadhati wrote:
I have seen "use the proxy table" answers on one too many occasions.
[snip]
So the question is this: is it worth requesting this as a feature in
5.1?

Probably yes (IMHO) but I suppose there are issues with that, otherwise it would have been a feature of Lua since a long time.

Note that, AFAIK, this proxy thingy isn't well documented (from a quick search on the Wiki). A search of [Lua "proxy table"] on Google gives very little hits, mostly on the mailing list...

To avoid searching in the mailing list archive, perhaps this item should go in the FAQ.

--
--=#=--=#=--=#=--=#=--=#=--=#=--=#=--=#=--=#=--
Philippe Lhoste (Paris -- France)
Professional programmer and amateur artist
http://Phi.Lho.free.fr
--=#=--=#=--=#=--=#=--=#=--=#=--=#=--=#=--=#=--

Reply | Threaded
Open this post in threaded view
|

Re: lua hacking wondering

Edgar Toernig
Philippe Lhoste wrote:
>
> Vijay Aswadhati wrote:
> > I have seen "use the proxy table" answers on one too many occasions. 
> [snip]
> > So the question is this: is it worth requesting this as a feature in
> > 5.1?
> 
> Probably yes (IMHO) but I suppose there are issues with that, otherwise 
> it would have been a feature of Lua since a long time.

It has been in Lua for a very long time (gettable/settable tag methods).
It was removed in Lua 5, possibly to get the number of meta-methods
for indexing down to two (instead of four).

IMHO, one of the two unfortunate changes in Lua 5 (the other one being
the handling of globals, setfenv & co).

Ciao, ET.

Reply | Threaded
Open this post in threaded view
|

Re: lua hacking wondering

Jamie Webb-3
In reply to this post by Philippe Lhoste
On Monday 21 June 2004 09:25, Philippe Lhoste wrote:
> Vijay Aswadhati wrote:
> > I have seen "use the proxy table" answers on one too many occasions.
>
> [snip]
>
> > So the question is this: is it worth requesting this as a feature in
> > 5.1?
>
> Probably yes (IMHO) but I suppose there are issues with that, otherwise
> it would have been a feature of Lua since a long time.

The issue is most likely one of speed. The current behaviour is invoked only 
for 'tables misses', which are presumed rare. The behaviour you are 
suggesting would require an additional lookup for every table access, 
regardless of whether the table actually has this metamethod. The same 
problem applies for a __next metamethod, which is one I'd certainly find 
useful.

The solution, I believe, is to support the __next metamethod for userdatas 
only, and then add a 'proxy userdata' constructor to the standard library. 
That would allow fully transparent proxies with reasonable efficiency, and 
obviously they would then be in the reference manual.

Of course, I should mention that proxy tables are included in Roberto's book.

-- Jamie Webb

Reply | Threaded
Open this post in threaded view
|

Re: lua hacking wondering

Dimitris Papavasileiou
Jamie Webb wrote:

The issue is most likely one of speed. The current behaviour is invoked only for 'tables misses', which are presumed rare. The behaviour you are suggesting would require an additional lookup for every table access, regardless of whether the table actually has this metamethod. The same problem applies for a __next metamethod, which is one I'd certainly find useful.

This is the impression I'm under as well, but I was wondering: since these entries in the metatable (the metamethods that is) are used by the VM it wouldn't be too ugly to keep a number of flags with each table/userdata for storing wich (of the "standard") metamethods have been set in it's metatable. Then every time, say, an index operation takes place all you have to do is check if the index flag is set (a very cheap operation) instead of looking up the metatable and indexing it. The flags can either be kept with the userdata/tables or their metatables depends on how tables/metatables are handled internally (if it's fast to get a table/userdata's metatable or if it's fast to get a metatable's table/userdata). Of course there are bound to be complications with this (like the fact that a metatable can be bound to more than one tables/userdatas), that I'm unaware of, as I know almost nothing about lua internals but I was just wondering if a solution in that direction might be possible as __settable, __gettable, __next metamethods would be very nice. Proxy tables seem to make lua API C code quite complex (you have to keep references to proxy tables, etc.) and besides it would be more logical to have these metamethods (if they could be implemented cheaply) than not to.

Dimitris P.

Reply | Threaded
Open this post in threaded view
|

Re: lua hacking wondering

Adam D. Moss
Dimitris Papavasiliou wrote:
This is the impression I'm under as well, but I was wondering: since these entries in the metatable (the metamethods that is) are used by the VM it wouldn't be too ugly to keep a number of flags with each table/userdata for storing wich (of the "standard") metamethods have been set in it's metatable. Then every time, say, an index operation takes place all you have to do is check if the index flag is set (a very cheap operation) instead of looking up the metatable and indexing it.

Actually I think this is roughly how it's implemented in reality.

--Adam
--
Adam D. Moss   . ,,^^   [hidden email]   http://www.foxbox.org/   co:3

Reply | Threaded
Open this post in threaded view
|

Re: lua hacking wondering

Roberto Ierusalimschy
> > This is the impression I'm under as well, but I was wondering:      
> > since these entries in the metatable (the metamethods that is)      
> > are used by the VM it wouldn't be too ugly to keep a number of      
> > flags with each table/userdata for storing wich (of the "standard") 
> > metamethods have been set in it's metatable. Then every time, say,  
> > an index operation takes place all you have to do is check if the   
> > index flag is set (a very cheap operation) instead of looking up    
> > the metatable and indexing it.                                      

> Actually I think this is roughly how it's implemented in reality.

The metatable itself has these bits. When we modify a table there
is no way to know which objects use it as a metatable, so there would
be no way to reset their bits.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

fast metamethods (was: lua hacking wondering)

Thiago Bastos-3
In reply to this post by Adam D. Moss
> > This is the impression I'm under as well, but I was wondering: since
> > these entries in the metatable (the metamethods that is) are used by the
> > VM it wouldn't be too ugly to keep a number of flags with each
> > table/userdata for storing wich (of the "standard") metamethods have
> > been set in it's metatable. Then every time, say, an index operation
> > takes place all you have to do is check if the index flag is set (a very
> > cheap operation) instead of looking up the metatable and indexing it.
> 
> Actually I think this is roughly how it's implemented in reality.

This is done but not for all metamethods. Only a subset (the most common
metamethods) is cached. Moreover, IIRC this scheme only optimizes 'misses',
that is, the flags records when a metamethod does NOT exist.

I have developed an automatic C++ binding generator (similar to toLua,
although its mechanics and implementation are completely different). It's
targeted at time-critical/real-time applications, such as games; designed
for systems where the core functionality is programmed in C++ and the
application logic is scripted in Lua.

Since my bindings make intensive use of metamethods, I had to optimize the
way metamethods interface with the C API.

My solution was only for full userdatas (not tables), which is what my games
needed; it's 100% backward compatible, but requires the use of some internal
Lua functions.

By default, full userdatas keep a reference to their metatables:

struct {
  ...
  struct Table *metatable;
  ...
} Udata;

What I have done is introduce a new field in the structure, which is used
instead of the metatable for invoking metamethods:

struct {
  ...
  struct Table *metatable;
  const TObject *fastTM;
  ...
} Udata;

The field 'fastTM' is an array of Lua objects, which is indexed from
TM_INDEX to TM_CALL (constants defined in ltm.h). Each table entry can be
set to point to a C closure, for example.

Then I changed the way metamethods are invoked, for full userdatas, in
ltm.c. Originally:

case LUA_TUSERDATA:
  return luaH_getstr(uvalue(o)->uv.metatable, ename);

Patched (lookup is made at the fastTM table first, if it is available):

case LUA_TUSERDATA:
  if( uvalue(o)->uv.fastTM ) {
    /* if a fastTM table is available, use it instead of the metatable */
    const TObject *method = &uvalue(o)->uv.fastTM[event];
    return ( method->tt == LUA_TNONE ? &luaO_nilobject : method );
  }
  return luaH_getstr(uvalue(o)->uv.metatable, ename);


My binding generator transparently generates internal Lua structures and
calls that make use of this fastTM mechanism, to generate much faster and
lighter bindings. The user himself knows nothing about it, so to port my
games to new Lua versions I only have to change the binding generator.

I believe that, if polished, this mechanism could be incorporated into the
official Lua API, as a way to optimize metamethods for full-userdatas.

I believe that 90% of the time, programmers create metatables for their
userdatas just to define metamathods (and nothing else). This patch allows
an alternative and much more efficient way of defining these metamethods.

By the way, the addition of the fastTM field in the Udata structure does not
impose additional memory usage, since that structure had some wasted space
available (due to alignment/union issues). So this functionality comes
virtually for free, I guess.

I have attached a diff file for the patch (against Lua 5.0.1).

PS: I plan to release my binding generator under an open-source license as
soon as I have some spare time. Game developers might find it useful.

-- Thiago Bastos

Attachment: fastTM.patch
Description: Binary data

Reply | Threaded
Open this post in threaded view
|

Re: lua hacking wondering

Dimitris Papavasileiou
In reply to this post by Roberto Ierusalimschy
Roberto Ierusalimschy wrote:

The metatable itself has these bits. When we modify a table there
is no way to know which objects use it as a metatable, so there would
be no way to reset their bits.
Wouldn't it be possible to store a pointer the the internal table structure of the metatable that is ssigned to each table/udata? That way the cost of checking wether a metamethod exists would be one pointer lookup and some bit testing wich is negligible (plus maybe some overhead in lua_setmetatable), and we would be able to have our precious __settable ,__gettable and maybe __next metamethods.

-- Dimitris

Reply | Threaded
Open this post in threaded view
|

Re: fast metamethods (was: lua hacking wondering)

Roberto Ierusalimschy
In reply to this post by Thiago Bastos-3
> This patch allows an alternative and much more efficient way of
> defining these metamethods.

Have you measured the speed up in "real-like" code?


> By the way, the addition of the fastTM field in the Udata structure
> does not impose additional memory usage, since that structure had some
> wasted space available (due to alignment/union issues).

Are you sure? In my machine, the original structure had 16 bytes, the
new one has 20 (which would go to 24 with double alignment).

-- Roberto

Reply | Threaded
Open this post in threaded view
|

RE: fast metamethods (was: lua hacking wondering)

Virgil Smith
So why not kill two birds with one stone.
Disallow metamethods on "simple" tables entirely.  Add a new type to the
language for tables that can have metatables.  Tables with metatables (like
full userdata) represent user defined types.

The requested metamethods could then be added without risking performance
degradation to tables in general (which is quite possibly a huge performance
threat to Lua as a whole).

OK, here's "the second bird".  If this new "extended table"(or whatever)
type is given a C-API accessible void* then it can replace full userdata and
simultaneously solve the per-userdata Lua-side value problem with no
"weak-table issues" involved.

I personally only see one caveat, and that is the globals table /
environments.  The problem with them is that they are very frequently
accessed, but also setting metamethods for them is extremely useful.

Well OK a second caveat is that I disagree with the authors concerning Lua
allocating memory for userdata's and so they might prefer that the C-side
accessible void* be a bit more than that in order to enable garbage
collection of some allocated space.  However, that detail is (I think)
irrelevant to the suggestion.




-----Original Message-----
From: [hidden email]
[[hidden email] Behalf Of Roberto
Ierusalimschy
Sent: Wednesday, June 23, 2004 6:41 AM
To: Lua list
Subject: Re: fast metamethods (was: lua hacking wondering)


> This patch allows an alternative and much more efficient way of
> defining these metamethods.

Have you measured the speed up in "real-like" code?


> By the way, the addition of the fastTM field in the Udata structure
> does not impose additional memory usage, since that structure had some
> wasted space available (due to alignment/union issues).

Are you sure? In my machine, the original structure had 16 bytes, the
new one has 20 (which would go to 24 with double alignment).

-- Roberto



12