Interpreter not building correct byte code

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Interpreter not building correct byte code

Tom Bates

>> At the moment, the interpreter does not build correct code from source

> 

>What errors does the compiler give?

> 

>> but the engine does execute compiled code correctly.

> 

>You mean, code compiled with luac?

>--lhf

 

Here’s a portion of my test code:

 

    char test10[] = "a=1;b=2;c=a+b;print(c)";

    if ((error = luaL_loadbuffer(L, test10, strlen(test10), "<test10>")) == 0) {

        error = lua_pcall(L, 0, 0, 0);

    }

    if (error != 0) {

        Syslog(loglist,LOG_SEVERE,"Err in test10: %s",lua_tostring(L, -1));

        lua_pop(L, 1);          // pop error message from the stack

    }

 

Running on the prototype hardware (i.e.68331), the compiler/interpreter generates no errors, i.e. luaL_loadbuffer returns 0. However, lua_pcall then either causes a fault or returns a bogus error message like “attempt to perform arithmetic on a string value”. To track this down, I worked my way down into the VM code and found that the instructions there do not match what is generated by the luac compiler running in Visual Studio 6 under Windows 2000 Pro. The first few instructions look good, but the rest is no good (e.g. no return instruction anywhere).

 

I can compile this lua source on the PC, convert the binary chunk into a C byte string, build that into the program, and it executes correctly a few times.

 

BTW, is there a good porting guide somewhere? I need something for what I call an “embedded” environment; i.e. not embedded in other software, but embedded in a hardware device with limited resources. Also, I’m working with version 5.0.2.

 

Thanks very much for the help!

Tom

 

Reply | Threaded
Open this post in threaded view
|

Re: Interpreter not building correct byte code

Ralph Hempel

BTW, is there a good porting guide somewhere? I need something for what I call an “embedded” environment; i.e. not embedded in other software, but embedded in a hardware device with limited resources. Also, I’m working with version 5.0.2.

Tom,

I'm very interested in the work you are doing, as I'm busy
porting 5.1.1 onto an ARM7 with 256K flash and 64K RAM.

I finally figured out one of my problems which was that
my memory allocator was not forcing the block size to an
aligned value.

I'm not sure about the 68331 but it might require aligned
access too.

There are a lot of reallocs when Lua runs, and I'm worried about
heap fragmentation. I need to instrument it to make sure that
it's not a bigger problem than I think it is, though...

I agree a porting guide would be handy, maybe that would
make an interesting Lua GEM?

Ralph

Reply | Threaded
Open this post in threaded view
|

RE: Interpreter not building correct byte code

Tom Bates
Ralph,

After walking all the way through the lexical parsing on both the PC (which
was working properly) and the prototype hardware, I've since found that
there's a bug in our realloc function! <gasp> Not to mention a minor
inefficiency where if both the old size and new size are the same, it still
reallocates the block!

I worked around the bug temporarily in luaM_realloc by using
malloc,copy,free; that seems to work, and the interpreter is now functioning
on the prototype.

The memory allocator we're using does return aligned blocks. I would think
that's a real necessity in any environment. Does the C standard have
anything to say about this? The 68331 will get an odd address trap, but
natural boundaries aren't required (i.e. 4-byte alignment for longs) since
it's basically a 16-bit machine externally.

Since the allocations all appear to be rather small, I don't think we're
going to have any issues with memory allocation. It crossed my mind to
instrument the allocator as well.

What's a Lua GEM? When the code seems stable, perhaps I'll post my thoughts
to get a "porting guide" started.

Thanks for the reply
Tom

-----Original Message-----
From: [hidden email]
[[hidden email]] On Behalf Of Ralph Hempel
Sent: Saturday, December 23, 2006 9:31 AM
To: Lua list
Subject: Re: Interpreter not building correct byte code


> BTW, is there a good porting guide somewhere? I need something for what 
> I call an "embedded" environment; i.e. not embedded in other software, 
> but embedded in a hardware device with limited resources. Also, I'm 
> working with version 5.0.2.

Tom,

I'm very interested in the work you are doing, as I'm busy
porting 5.1.1 onto an ARM7 with 256K flash and 64K RAM.

I finally figured out one of my problems which was that
my memory allocator was not forcing the block size to an
aligned value.

I'm not sure about the 68331 but it might require aligned
access too.

There are a lot of reallocs when Lua runs, and I'm worried about
heap fragmentation. I need to instrument it to make sure that
it's not a bigger problem than I think it is, though...

I agree a porting guide would be handy, maybe that would
make an interesting Lua GEM?

Ralph


Reply | Threaded
Open this post in threaded view
|

A BIG Thank You and Holiday Wishes

Tom Bates

I’d like to take this time to thank the people who devote their time to helping everyone out.

 

Thank you, Luiz, for your efforts – they are much appreciated. Thanks to the Lua developers for an extraordinary language.

 

I’d also like to extend my best holiday wishes to everyone!

 

Merry Christmas!

Tom

 

Reply | Threaded
Open this post in threaded view
|

Re: Interpreter not building correct byte code

Ralph Hempel
In reply to this post by Tom Bates
Tom Bates wrote:
Ralph,

After walking all the way through the lexical parsing on both the PC (which
was working properly) and the prototype hardware, I've since found that
there's a bug in our realloc function! <gasp> Not to mention a minor
inefficiency where if both the old size and new size are the same, it still
reallocates the block!

My problem was with realloc too. When reducing the size of a block
of memory to a misaligned value, the pointers in the reallocated block
pointed to a misaligned value for the next block.

Increasing the block size worked fine because it called malloc() which
forced a properly aligned block.

I worked around the bug temporarily in luaM_realloc by using
malloc,copy,free; that seems to work, and the interpreter is now functioning
on the prototype.

The memory allocator we're using does return aligned blocks. I would think
that's a real necessity in any environment. Does the C standard have
anything to say about this? The 68331 will get an odd address trap, but
natural boundaries aren't required (i.e. 4-byte alignment for longs) since
it's basically a 16-bit machine externally.

I don't think the C standard says anything about this, but I'd need
to check. Aligning things to 16 bit boundaries will avoid extra
data fetches, of course. I think you want things aligned as much
as possible for speed.

Since the allocations all appear to be rather small, I don't think we're
going to have any issues with memory allocation. It crossed my mind to
instrument the allocator as well.

It's the reallocs I'm worried about. Lets say when Lua parses some
code it wants a block of 32 bytes, and then it reduces the block to
8 bytes. Then we had better hope that the 24 byte block (less
attributes) can be used or merged with another subsequent block,
or we end up with pretty gaping holes in the memory space.

What's a Lua GEM? When the code seems stable, perhaps I'll post my thoughts
to get a "porting guide" started.

Search the group. Roberto and the others are thinking of putting
together a little book of short Gems, ala Bentley's "Programming
Gems" Putting Lua on embedded processors seems like a great topic!

Cheers, Ralph

Reply | Threaded
Open this post in threaded view
|

Re: Interpreter not building correct byte code

Mark Edgar
In reply to this post by Tom Bates
Tom Bates wrote:
The memory allocator we're using does return aligned blocks. I would think
that's a real necessity in any environment. Does the C standard have
anything to say about this?

Yes; in ISO 9899:1999 7.20.3 p1

"The pointer returned if the allocation succeeds is suitably aligned so that it may be assigned to a pointer to any type of object and then used to access such an object or an array of such objects in the space allocated (until the space is explicitly deallocated)."

					-Mark

Reply | Threaded
Open this post in threaded view
|

RE: Interpreter not building correct byte code

Tom Bates
Excellent! Thanks, Mark!

-----Original Message-----
From: [hidden email]
[[hidden email]] On Behalf Of Mark Edgar
Sent: Saturday, December 23, 2006 2:56 PM
To: Lua list
Subject: Re: Interpreter not building correct byte code

Tom Bates wrote:
> The memory allocator we're using does return aligned blocks. I would think
> that's a real necessity in any environment. Does the C standard have
> anything to say about this?

Yes; in ISO 9899:1999 7.20.3 p1

"The pointer returned if the allocation succeeds is suitably aligned so 
that it may be assigned to a pointer to any type of object and then used 
to access such an object or an array of such objects in the space 
allocated (until the space is explicitly deallocated)."

					-Mark

-- 
No virus found in this incoming message.
Checked by AVG Free Edition.
Version: 7.5.430 / Virus Database: 268.15.26/598 - Release Date: 12/22/2006
3:22 PM
 


Reply | Threaded
Open this post in threaded view
|

Re: A BIG Thank You and Holiday Wishes

Luiz Henrique de Figueiredo
In reply to this post by Tom Bates
> Thank you, Luiz, for your efforts - they are much appreciated. Thanks to the
> Lua developers for an extraordinary language.

Thank you and all the Lua community for being so loyal and inspiring.
Happy Holidays to everyone!
--lhf

Reply | Threaded
Open this post in threaded view
|

RE: A BIG Thank You and Holiday Wishes

Richard Ranft
In reply to this post by Tom Bates
Here here!  You guys on the Lua team have done a magnificent job over the years and I personally am very grateful.  Thank you all and Merry Christmas!
 
Rich
-----Original Message-----
From: [hidden email] [mailto:[hidden email]]On Behalf Of Tom Bates
Sent: Saturday, December 23, 2006 8:48 AM
To: 'Lua list'
Subject: A BIG Thank You and Holiday Wishes

I’d like to take this time to thank the people who devote their time to helping everyone out.

 

Thank you, Luiz, for your efforts – they are much appreciated. Thanks to the Lua developers for an extraordinary language.

 

I’d also like to extend my best holiday wishes to everyone!

 

Merry Christmas!

Tom

 

Reply | Threaded
Open this post in threaded view
|

Re: Interpreter not building correct byte code

David Jones-2
In reply to this post by Tom Bates

On 22 Dec 2006, at 18:32, Tom Bates wrote:

>> At the moment, the interpreter does not build correct code from source

>

>What errors does the compiler give?

	

Running on the prototype hardware (i.e.68331), the compiler/ interpreter generates no errors, i.e. luaL_loadbuffer returns 0. However, lua_pcall then either causes a fault or returns a bogus error message like “attempt to perform arithmetic on a string value”. To track this down, I worked my way down into the VM code and found that the instructions there do not match what is generated by the luac compiler running in Visual Studio 6 under Windows 2000 Pro. The first few instructions look good, but the rest is no good (e.g. no return instruction anywhere).

The bytecodes produced by luac are endian dependent (as well as dependent on word-width and floating point format).

An Intel PC is little-endian, a 68K is big-endian. You can either swab offline, using Kein-Hong Man's ChunkSpy http://luaforge.net/ projects/chunkspy/ , or rewrite lundump.c to use the other (or either) endianness. I think LHF posted a modified lundump.c to the list last year.

This useful fact about luac's bytecodes is documented in its manual: http://www.lua.org/manual/5.1/luac.html

I hope that helps.  Good look with re-alloc.

Cheers,
 drj




Reply | Threaded
Open this post in threaded view
|

RE: Interpreter not building correct byte code

Tom Bates

> The bytecodes produced by luac are endian dependent (as well as

> dependent on word-width and floating point format).

>

> An Intel PC is little-endian, a 68K is big-endian.  You can either

> swab offline, using Kein-Hong Man's ChunkSpy http://luaforge.net/

> projects/chunkspy/ , or rewrite lundump.c to use the other (or

> either) endianness.  I think LHF posted a modified lundump.c to the

> list last year.

>

> This useful fact about luac's bytecodes is documented in its manual:

> http://www.lua.org/manual/5.1/luac.html

>

> I hope that helps.  Good look with re-alloc.

>

> Cheers,

>   drj

 

The problem I was experiencing was 100% due to a bug in our realloc function. Now that that is cleared up, I am able to compile scripts under Visual Studio 6 on an Intel PC (low-endian) and run them unmodified on the prototype hardware using a 68331 (high-endian). There have been no apparent endianness issues (so far). Perhaps the 5.0.2 lundump is already using that LHF code?

 

By increasing the stack space, I was able to get the parser working as well. I have now split out the parsing code thanks to a tech note on lua.org (although it was for version 4.0) and made the code into libraries that allow me to build either barebones (no parser; compiled scripts only), scripting (with parser), and full-blown (with interpreter).

 

Thanks

Tom

 

Reply | Threaded
Open this post in threaded view
|

Re: Interpreter not building correct byte code

Rici Lake-2

On 1-Jan-07, at 6:05 PM, Tom Bates wrote:

The problem I was experiencing was 100% due to a bug in our realloc function. Now that that is cleared up, I am able to compile scripts under Visual Studio 6 on an Intel PC (low-endian) and run them unmodified on the prototype hardware using a 68331 (high-endian). There have been no apparent endianness issues (so far). Perhaps the 5.0.2 lundump is already using that LHF code?

5.0.2 does byte-swapping; it was removed from 5.1, although it's easy enough to add back in.

 
By increasing the stack space, I was able to get the parser working as well. I have now split out the parsing code thanks to a tech note on lua.org (although it was for version 4.0) and made the code into libraries that allow me to build either barebones (no parser; compiled scripts only), scripting (with parser), and full-blown (with interpreter).