Lua with Paged / Banked memory architecture

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Lua with Paged / Banked memory architecture

Alexander Goldstone
Due to restrictions in a microcontroller, we have 1MB available but can
only address it in 16KB chunks.

We have successfully ported Lua but are running out of memory due to
this limitation so our thought is to have Lua use a 'paged' or 'banked'
memory scheme (i.e. addressing all of the 1MB by switching between pages).

Has anybody attempted this kind of alteration and/or can you point us in
the right direction as to how we may get started with this if indeed the
Lua source would even allow it without a substantial re-write?

Thanks in advance.

Alex.
Reply | Threaded
Open this post in threaded view
|

Re: Lua with Paged / Banked memory architecture

Ralph Hempel
Alexander Goldstone wrote:

> Due to restrictions in a microcontroller, we have 1MB available but can
> only address it in 16KB chunks.
>
> We have successfully ported Lua but are running out of memory due to
> this limitation so our thought is to have Lua use a 'paged' or 'banked'
> memory scheme (i.e. addressing all of the 1MB by switching between pages).
>
> Has anybody attempted this kind of alteration and/or can you point us in
> the right direction as to how we may get started with this if indeed the
> Lua source would even allow it without a substantial re-write?

Is the banking in ROM or RAM? And 16k? That's an awfully small window.

Can you share a few more details?

Ralph
Reply | Threaded
Open this post in threaded view
|

Re: Lua with Paged / Banked memory architecture

BogdanM
In reply to this post by Alexander Goldstone
You could _probably_ implement a custom allocator and modify the VM to switch to a certain bank when loading a Lua object, but 16k seems to be a very small window. Don't know how much you can do with it.

Best,
Bogdan

On Fri, Mar 27, 2009 at 6:14 PM, Alexander Goldstone <[hidden email]> wrote:
Due to restrictions in a microcontroller, we have 1MB available but can
only address it in 16KB chunks.

We have successfully ported Lua but are running out of memory due to
this limitation so our thought is to have Lua use a 'paged' or 'banked'
memory scheme (i.e. addressing all of the 1MB by switching between pages).

Has anybody attempted this kind of alteration and/or can you point us in
the right direction as to how we may get started with this if indeed the
Lua source would even allow it without a substantial re-write?

Thanks in advance.

Alex.

Reply | Threaded
Open this post in threaded view
|

Re: Lua with Paged / Banked memory architecture

Alexander Goldstone
In reply to this post by Ralph Hempel


Ralph Hempel wrote:
>
> Is the banking in ROM or RAM? And 16k? That's an awfully small window.
>
> Can you share a few more details?
>
> Ralph

The banking is in external RAM.

We have sufficient code space as the microcontroller is capable of
addressing 16MB of it - (Harvard architecture so separate address and
data buses).

It's main limitation is on data space, since it can only address 64KB of
ram for which 32KB is on chip for application variable storage, 16KB is
used for constant data access and the remaining 16KB would be for page
access to the external 1MB of data ram that would only be used to hold
the Lua scripts.

Thanks,

Alex
Reply | Threaded
Open this post in threaded view
|

Re: Lua with Paged / Banked memory architecture

Asko Kauppi

Did I understand this right: you'd only have the Lua
_scripts_ (precompiled bytecode) in the banked memory?
  How much of scripts do you have?

That would mean no banking is used for the GC'able
objects. Makes no sense to me..

-asko


On Fri, 27 Mar 2009 17:35:48 +0000
  Alexander Goldstone <[hidden email]> wrote:

>
>
> Ralph Hempel wrote:
>>
>> Is the banking in ROM or RAM? And 16k? That's an awfully
>>small window.
>>
>> Can you share a few more details?
>>
>> Ralph
>
> The banking is in external RAM.
>
> We have sufficient code space as the microcontroller is
>capable of
> addressing 16MB of it - (Harvard architecture so
>separate address and
> data buses).
>
> It's main limitation is on data space, since it can only
>address 64KB of
> ram for which 32KB is on chip for application variable
>storage, 16KB is
> used for constant data access and the remaining 16KB
>would be for page
> access to the external 1MB of data ram that would only
>be used to hold
> the Lua scripts.
>
> Thanks,
>
> Alex

Reply | Threaded
Open this post in threaded view
|

Re: Lua with Paged / Banked memory architecture

Alexander Goldstone


[hidden email] wrote:

>
> Did I understand this right: you'd only have the Lua _scripts_
> (precompiled bytecode) in the banked memory?  How much of scripts do you
> have?
>
> That would mean no banking is used for the GC'able objects. Makes no
> sense to me..
>
> -asko
>
>

Sorry, I'll try to be clearer:

+ The microcontroller has enough internal flash memory for our entire C
application (excluding the Lua script).

+ The microcontroller is based on the Harvard architecture which means
it addresses code and data space separately.

+ There is enough code-space for the C application (including the Lua
libraries and basic interpreter).

+ There is enough data-space for variables and constants associated with
the C application.

+ We can reference an additional 16KB data-space. The address of this
16KB block can be changed on the fly to reference anywhere across our
external 1MB RAM.

+ The microcontroller is connected to external NAND flash storage. The
only way in which this is relevant is that we are storing our script
files in this external NAND flash and loading it in at runtime.

+ All of this works. If, for example, our Lua script performs a simple
calculation and prints out the result then this works fine.

+ As our Lua script grows, it runs out of memory.

+ Our thinking is that we could give Lua 1MB to run its scripts (instead
of the limited data-space it currently has) if we could alter lua to use
16KB pages/banks. We only have to update the value of a single register
to re-point the useable 16KB block to another area of the 1MB. So what
we need to do (if possible without a substantial re-write of Lua) is to
have Lua allocate and reference memory accross multiple pages and switch
pages accordingly.


Does that make more sense than my previous email?

Thanks,

Alex.

> On Fri, 27 Mar 2009 17:35:48 +0000
>  Alexander Goldstone <[hidden email]> wrote:
>>
>>
>> Ralph Hempel wrote:
>>>
>>> Is the banking in ROM or RAM? And 16k? That's an awfully small window.
>>>
>>> Can you share a few more details?
>>>
>>> Ralph
>>
>> The banking is in external RAM.
>>
>> We have sufficient code space as the microcontroller is capable of
>> addressing 16MB of it - (Harvard architecture so separate address and
>> data buses).
>>
>> It's main limitation is on data space, since it can only address 64KB of
>> ram for which 32KB is on chip for application variable storage, 16KB is
>> used for constant data access and the remaining 16KB would be for page
>> access to the external 1MB of data ram that would only be used to hold
>> the Lua scripts.
>>
>> Thanks,
>>
>> Alex
>
Reply | Threaded
Open this post in threaded view
|

Re: Lua with Paged / Banked memory architecture

Ralph Hempel
Alexander Goldstone wrote:

<snip>

> + Our thinking is that we could give Lua 1MB to run its scripts (instead
> of the limited data-space it currently has) if we could alter lua to use
> 16KB pages/banks. We only have to update the value of a single register
> to re-point the useable 16KB block to another area of the 1MB. So what
> we need to do (if possible without a substantial re-write of Lua) is to
> have Lua allocate and reference memory accross multiple pages and switch
> pages accordingly.

<Puts on glasses and strokes chin thoughfully>

Way back in the old days, banked memory architectures were common, and
some compiler vendors even supported this notion for variables
and function entry points.

You would put all the "fast" access variables and code into memory that
could never be banked out, and declared others as banked.

Basically, a function call or variable reference that was allocated
in the banked space generated special code that switched to the
desired bank and it all just "worked". Unless it didn't and then
you had to mess with your linker until it did.

This only works for things that you know ahead of time are going
to be banked, and I doubt any modern compiler supports this at all.

I don't think rewriting Lua is going to get you much bang for your
buck, the problem needs support right down at the compiler level.

16K is awfully small for a Lua script. Not the code mind you, but the
environment that is set up every time you initialize a Lua state.

Depending on how big your script is, it might fit. But unless you work
very hard to recover the space used by that script (by nilling)
variables and then forcing garbage collection, you'll end up
with very little space due to heap fragmentation.

Can you just fully reinitialize the Lua state every time
you run a new script?

And have you given any consideration to Forth as a scripting
language for your very tight memory constraints?

Ralph
Reply | Threaded
Open this post in threaded view
|

Re: Lua with Paged / Banked memory architecture

Asko Kauppi
In reply to this post by Alexander Goldstone

To me it would be trivial to start doing this as:

- make a custom allocator that handles the GC'able memory accesses,  
giving the "pointer" value as bank + offset.
- then any accesses will need to be guaranteed to switch in the right  
bank (needs modifying Lua core)

But the hard part is, as Ralph mentions, the 'lua_State' things  
itself, or whatever gets birth when bytecode is read in.

You should make some limitations as to what you actually want to run.  
Making that architecture run "anything" (any size of script, with any  
size of GC'able objects, tables of any size, etc...) is most likely  
not going to happen. And in that case, you really want to _know_ the  
limitations and not stumble upon them at one point, later on.

I would simply move to a better MPU, and feel safe.  At least join the  
elua mailing lists, and observe the advancements there.

-asko


Alexander Goldstone kirjoitti 28.3.2009 kello 15:22:

>
>
> [hidden email] wrote:
>>
>> Did I understand this right: you'd only have the Lua _scripts_
>> (precompiled bytecode) in the banked memory?  How much of scripts  
>> do you
>> have?
>>
>> That would mean no banking is used for the GC'able objects. Makes no
>> sense to me..
>>
>> -asko
>>
>>
>
> Sorry, I'll try to be clearer:
>
> + The microcontroller has enough internal flash memory for our  
> entire C
> application (excluding the Lua script).
>
> + The microcontroller is based on the Harvard architecture which means
> it addresses code and data space separately.
>
> + There is enough code-space for the C application (including the Lua
> libraries and basic interpreter).
>
> + There is enough data-space for variables and constants associated  
> with
> the C application.
>
> + We can reference an additional 16KB data-space. The address of this
> 16KB block can be changed on the fly to reference anywhere across our
> external 1MB RAM.
>
> + The microcontroller is connected to external NAND flash storage. The
> only way in which this is relevant is that we are storing our script
> files in this external NAND flash and loading it in at runtime.
>
> + All of this works. If, for example, our Lua script performs a simple
> calculation and prints out the result then this works fine.
>
> + As our Lua script grows, it runs out of memory.
>
> + Our thinking is that we could give Lua 1MB to run its scripts  
> (instead
> of the limited data-space it currently has) if we could alter lua to  
> use
> 16KB pages/banks. We only have to update the value of a single  
> register
> to re-point the useable 16KB block to another area of the 1MB. So what
> we need to do (if possible without a substantial re-write of Lua) is  
> to
> have Lua allocate and reference memory accross multiple pages and  
> switch
> pages accordingly.
>
>
> Does that make more sense than my previous email?
>
> Thanks,
>
> Alex.
>
>> On Fri, 27 Mar 2009 17:35:48 +0000
>> Alexander Goldstone <[hidden email]> wrote:
>>>
>>>
>>> Ralph Hempel wrote:
>>>>
>>>> Is the banking in ROM or RAM? And 16k? That's an awfully small  
>>>> window.
>>>>
>>>> Can you share a few more details?
>>>>
>>>> Ralph
>>>
>>> The banking is in external RAM.
>>>
>>> We have sufficient code space as the microcontroller is capable of
>>> addressing 16MB of it - (Harvard architecture so separate address  
>>> and
>>> data buses).
>>>
>>> It's main limitation is on data space, since it can only address  
>>> 64KB of
>>> ram for which 32KB is on chip for application variable storage,  
>>> 16KB is
>>> used for constant data access and the remaining 16KB would be for  
>>> page
>>> access to the external 1MB of data ram that would only be used to  
>>> hold
>>> the Lua scripts.
>>>
>>> Thanks,
>>>
>>> Alex
>>

Reply | Threaded
Open this post in threaded view
|

Re: Lua with Paged / Banked memory architecture

Nick Gammon
In reply to this post by Alexander Goldstone
Hmmm, Lua on a PIC chip eh? Sounds interesting. Which chip is it, if  
you don't mind my asking?


One possible solution would be to simply implement the memory access  
via an extra function you add, eg.

PutIntoMemory (bank, string)  -- put up to 16 Kb string into a bank

and

GetFromMemory (bank, string)  -- get string back

Depending on what you are trying to achieve with the application, that  
might work. Or let the access to the banked memory be accessible by  
key (eg. by using an STL map type).

For example, if you are just trying to log a lot of numbers into the  
memory, you might do this:

PutIntoMemory (bank, key, number)


- Nick
Reply | Threaded
Open this post in threaded view
|

Re: Lua with Paged / Banked memory architecture

BogdanM
PICs you say? Maybe you could use this then:

http://www.lapsi.eletro.ufrgs.br/~socal/projects/picoLua/

Probably reasonably easy to adapt for your needs.

Best,
Bogdan

On Sat, Mar 28, 2009 at 11:08 PM, Nick Gammon <[hidden email]> wrote:
Hmmm, Lua on a PIC chip eh? Sounds interesting. Which chip is it, if you don't mind my asking?


One possible solution would be to simply implement the memory access via an extra function you add, eg.

PutIntoMemory (bank, string)  -- put up to 16 Kb string into a bank

and

GetFromMemory (bank, string)  -- get string back

Depending on what you are trying to achieve with the application, that might work. Or let the access to the banked memory be accessible by key (eg. by using an STL map type).

For example, if you are just trying to log a lot of numbers into the memory, you might do this:

PutIntoMemory (bank, key, number)


- Nick