Lua and Sol2 on Embedded Target?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Lua and Sol2 on Embedded Target?

Russell Haley
Hello,

So this isn't so much a question as a brain storming exercise:

We have a client that has an embedded system and they want to be able to control individual components, but they can't really do that with their production software. For instance, they want to be able to change voltages on an LED and then record the ADC outputs (or something like that) without running in the main state loop.  They want the flexibility to be able write scripts on the PC Host side that sends commands to the embedded system, which queues and executes them. hmmm....

The production firmware and all the drivers are written in C++ (some C at the bottom of the stack) built with the Keil compiler and the system uses the Keil RTX RTOS.  The hardware is an STM32F-something rather. It has an 120Mhz processor and 256K RAM... plus an external 1 MB (Mega BYTE) of RAM that they aren't using. I don't remember what the flash size is, but it's large as well. hmmm...

This really sounds like a job for Lua. 

My thought is to leave the RTX RTOS in place and add a Lua interpreter to the main thread where the state engine used to be (after I remove it). The application is written in C++ so I was thinking about using the sol2.hpp interface, which would allow me to use the built in C++ classes.

I envision it working like this: There would be two modes, command and script mode. Command mode would be akin to running the interactive interpreter. The user would type a command or chunk into a terminal on the host PC. The PC application would interpreted the chunk for the *target* architecture, then send it to the VM on the target to run and send back results. Script mode would be ... umm ... executing a script. In this case, the PC host application would interpret the entire script for the target, then send the entire byte code to be executed at once and possibly persisted. My current assumption is StdOut would be sent directly out the serial port.

So, now comes the torrent of questions:
- Has anyone used Lua with Keil?
- Has anyone used Sol2 with Keil? (And would the wise and mysterious Phantom Derp Storm support it?)
- With 1 Mb of Ram at my disposal, how much of Lua would be wise to put on the target? 
- Given the prior question, should I bother interpreting on the PC host?
- The Sol2 single file is 800Kb. Can Sol2 be chopped down? How small? What would be wise?
- Has anyone ever shipped byte code across a serial port? How bad an idea do you think that is?
...
Ad nauseam.

This is an open ended question so everything is on the table. I'm hoping to get ideas from anyone with experience doing this type of thing? I'll be going back over the mailing list archives as I know there have been a few about this very subject, but if you remember a specific thread, please send me a link? I'd also be interested in any libraries people think might be useful? 

I have no idea if the client (or even anyone at work) will go for this idea, but I think it's a winner. All input welcome.

Cheers,
Russ
Reply | Threaded
Open this post in threaded view
|

Re: Lua and Sol2 on Embedded Target?

Frank Kastenholz-2


> On Oct 24, 2019, at 10:41 PM, Russell Haley <[hidden email]> wrote:
>
> Hello,
>
> So this isn't so much a question as a brain storming exercise:...

Hi
Several years ago I was involved in a similar effort.  While I can’t talk about the technical details, I can say

1) this project should be quite doable.  The one I did took 2-3 developers less than 6 months.

2) Lua, as we all know is a very small footprint.  The libraries and interface code, on the other hand can use a lot of space. We had to write a fair amount of code to interface the Lua programs to the system
It was running on.   If I recall, the final executable image was between 5x and 10x the size of the base Lua.  We had some unique requirements that you might not have, but regardless, this is something to keep an eye on.

3) we also ran into the issue with the Lua application’s memory and processing needs - the app writers all had an attitude that “resources are free”. :-(

4) depending on the capabilities of the RTOS you may need to write a “timeslicer” for the Lua VM - have it execute N VM instructions, yield back to the RTOS, and then get control back to repeat the cycle.  This was probably the single hardest bit of code we had to do for our project (not that it was that hard).  We also had a requirement to be able to run multiple Lua programs independently (think separate processes in Unix)

Anyway, good luck and let us all know how it works

Frank Kastenholz




Reply | Threaded
Open this post in threaded view
|

Re: Lua and Sol2 on Embedded Target?

Russell Haley


On Fri, Oct 25, 2019 at 1:42 AM Frank Kastenholz <[hidden email]> wrote:


> On Oct 24, 2019, at 10:41 PM, Russell Haley <[hidden email]> wrote:
>
> Hello,
>
> So this isn't so much a question as a brain storming exercise:...

Hi
Several years ago I was involved in a similar effort.  While I can’t talk about the technical details, I can say

1) this project should be quite doable.  The one I did took 2-3 developers less than 6 months.

2) Lua, as we all know is a very small footprint.  The libraries and interface code, on the other hand can use a lot of space. We had to write a fair amount of code to interface the Lua programs to the system
It was running on.   If I recall, the final executable image was between 5x and 10x the size of the base Lua.  We had some unique requirements that you might not have, but regardless, this is something to keep an eye on.

3) we also ran into the issue with the Lua application’s memory and processing needs - the app writers all had an attitude that “resources are free”. :-(

4) depending on the capabilities of the RTOS you may need to write a “timeslicer” for the Lua VM - have it execute N VM instructions, yield back to the RTOS, and then get control back to repeat the cycle.  This was probably the single hardest bit of code we had to do for our project (not that it was that hard).  We also had a requirement to be able to run multiple Lua programs independently (think separate processes in Unix)

Anyway, good luck and let us all know how it works

Frank Kastenholz

Thanks Frank. 

How does one cross compile in Lua?

Russ
Reply | Threaded
Open this post in threaded view
|

Re: Lua and Sol2 on Embedded Target?

Russell Haley


On Fri, Oct 25, 2019 at 8:12 AM Russell Haley <[hidden email]> wrote:


On Fri, Oct 25, 2019 at 1:42 AM Frank Kastenholz <[hidden email]> wrote:


> On Oct 24, 2019, at 10:41 PM, Russell Haley <[hidden email]> wrote:
>
> Hello,
>
> So this isn't so much a question as a brain storming exercise:...

Hi
Several years ago I was involved in a similar effort.  While I can’t talk about the technical details, I can say

1) this project should be quite doable.  The one I did took 2-3 developers less than 6 months.

2) Lua, as we all know is a very small footprint.  The libraries and interface code, on the other hand can use a lot of space. We had to write a fair amount of code to interface the Lua programs to the system
It was running on.   If I recall, the final executable image was between 5x and 10x the size of the base Lua.  We had some unique requirements that you might not have, but regardless, this is something to keep an eye on.

3) we also ran into the issue with the Lua application’s memory and processing needs - the app writers all had an attitude that “resources are free”. :-(

4) depending on the capabilities of the RTOS you may need to write a “timeslicer” for the Lua VM - have it execute N VM instructions, yield back to the RTOS, and then get control back to repeat the cycle.  This was probably the single hardest bit of code we had to do for our project (not that it was that hard).  We also had a requirement to be able to run multiple Lua programs independently (think separate processes in Unix)

Anyway, good luck and let us all know how it works

Frank Kastenholz

Thanks Frank. 

How does one cross compile in Lua?

Russ
Okay, the threads of information are starting to come together:

Thanks LHF 
Reply | Threaded
Open this post in threaded view
|

Re: Lua and Sol2 on Embedded Target?

Russell Haley


On Fri, Oct 25, 2019 at 8:45 AM Russell Haley <[hidden email]> wrote:


On Fri, Oct 25, 2019 at 8:12 AM Russell Haley <[hidden email]> wrote:


On Fri, Oct 25, 2019 at 1:42 AM Frank Kastenholz <[hidden email]> wrote:


> On Oct 24, 2019, at 10:41 PM, Russell Haley <[hidden email]> wrote:
>
> Hello,
>
> So this isn't so much a question as a brain storming exercise:...

Hi
Several years ago I was involved in a similar effort.  While I can’t talk about the technical details, I can say

1) this project should be quite doable.  The one I did took 2-3 developers less than 6 months.

2) Lua, as we all know is a very small footprint.  The libraries and interface code, on the other hand can use a lot of space. We had to write a fair amount of code to interface the Lua programs to the system
It was running on.   If I recall, the final executable image was between 5x and 10x the size of the base Lua.  We had some unique requirements that you might not have, but regardless, this is something to keep an eye on.

3) we also ran into the issue with the Lua application’s memory and processing needs - the app writers all had an attitude that “resources are free”. :-(

4) depending on the capabilities of the RTOS you may need to write a “timeslicer” for the Lua VM - have it execute N VM instructions, yield back to the RTOS, and then get control back to repeat the cycle.  This was probably the single hardest bit of code we had to do for our project (not that it was that hard).  We also had a requirement to be able to run multiple Lua programs independently (think separate processes in Unix)

Anyway, good luck and let us all know how it works

Frank Kastenholz

Thanks Frank. 

How does one cross compile in Lua?

Russ
Okay, the threads of information are starting to come together:

Thanks LHF 

Sorry, I didn't mean to make it look like I am LHF, I meant to say thank you to LHF for the patch.

Cheers, 
Russell
Reply | Threaded
Open this post in threaded view
|

Re: Lua and Sol2 on Embedded Target?

Frank Kastenholz-2
In reply to this post by Russell Haley


> On Oct 25, 2019, at 11:12 AM, Russell Haley <[hidden email]> wrote:
>
> Thanks Frank.
>
> How does one cross compile in Lua?
>
> Russ

We didn’t
We downloaded the lua source to the target and compiled locally. a couple of reasons

1 The compiler is not that big

2 we were targeting heterogeneous systems, so we’d have to precompile for all targets (and possibly variants of each target)

3 we thought we wanted the ability to add additional code to an executing lua program  (which would compile it via load() and execute it).

Frank



Reply | Threaded
Open this post in threaded view
|

Re: Lua and Sol2 on Embedded Target?

Chris Smith
In reply to this post by Russell Haley

On 25 Oct 2019, at 06:36, Russell Haley <[hidden email]> wrote:
>
> This really sounds like a job for Lua.

Is it really? Do you have a requirement to use Lua? Don’t get me wrong, I’d love to see this work, but are you choosing the right tool for the job, or trying to make the job fit the tool?

There are a few projects out there using MicroPython on STM32 boards which work very well: Pycom, pyboard, etc. Unless you are set on using Lua, this may be a better approach for you.

Chris

Reply | Threaded
Open this post in threaded view
|

Re: Lua and Sol2 on Embedded Target?

Russell Haley


On Fri, Oct 25, 2019 at 9:41 AM Chris Smith <[hidden email]> wrote:

On 25 Oct 2019, at 06:36, Russell Haley <[hidden email]> wrote:
>
> This really sounds like a job for Lua.

Is it really? Do you have a requirement to use Lua? Don’t get me wrong, I’d love to see this work, but are you choosing the right tool for the job, or trying to make the job fit the tool?

There are a few projects out there using MicroPython on STM32 boards which work very well: Pycom, pyboard, etc. Unless you are set on using Lua, this may be a better approach for you.

Chris

Much appreciated, I didn't know this existed. Language is less of a concern, the idea is to allow for generalized scripting on the unit itself. I'll add micropython to the options list... Albeit down at the bottom of the list in tiny tiny tiny font. ;)

Russ
Reply | Threaded
Open this post in threaded view
|

Re: Lua and Sol2 on Embedded Target?

Philipp Janda
In reply to this post by Russell Haley
Am 25.10.19 um 04:41 schröbte Russell Haley:
> Hello,
>

Hi!

>
> So, now comes the torrent of questions:
> - Has anyone used Lua with Keil?

I haven't, but it seems llvm/clang based with complete c++14 support, so
you should be fine.

> - Has anyone used Sol2 with Keil? (And would the wise and mysterious
> Phantom Derp Storm support it?)

Same.

However, you can't completely disable exceptions when using Sol2, so you
should check whether exceptions are already enabled in your code base.

> - With 1 Mb of Ram at my disposal, how much of Lua would be wise to put on
> the target?

1 Mb RAM is plenty. You should go for 32 bit ints and floats though,
because this will halve your memory needs (pointers are 32 bit, too),
and you probably don't have 64bit float hardware anyway. Coroutines are
expensive, btw.

> - Given the prior question, should I bother interpreting on the PC host?
> - The Sol2 single file is 800Kb. Can Sol2 be chopped down? How small? What
> would be wise?

This is templated source code. Most of it won't end up in your image anyway.

> - Has anyone ever shipped byte code across a serial port? How bad an idea
> do you think that is?

You will need a custom luac on the host side (which is a constant source
of error and confusion if you have native Lua installed as well), and
you can forget sandboxing on the embedded device. The parser is not that
big, so I wouldn't bother.

>
> Cheers,
> Russ
>


Philipp



Reply | Threaded
Open this post in threaded view
|

Re: Lua and Sol2 on Embedded Target?

JeanHeyd Meneide
Dear Russ,

On Sat, Oct 26, 2019 at 10:54 AM Philipp Janda <[hidden email]> wrote:

>
> Am 25.10.19 um 04:41 schröbte Russell Haley:
> > Hello,
> >
>
> Hi!
>
> >
> > So, now comes the torrent of questions:
> > - Has anyone used Lua with Keil?
>
> I haven't, but it seems llvm/clang based with complete c++14 support, so
> you should be fine.
>
> > - Has anyone used Sol2 with Keil? (And would the wise and mysterious
> > Phantom Derp Storm support it?)
>
> Same.
>
     sol2 with without RTTI and without exceptions (since way back in
the early versions of v2), so you should be fine there. Let me know if
something breaks if you turn exceptions off, and make sure to read:
https://sol2.readthedocs.io/en/latest/safety.html

     Only thing I caution you over is gratuitous use of lambdas and
such for function binding: prefer writing member functions / free
functions and binding those directly, since they have the smallest
binary footprint.

     I am in the process of working on some space-saving optimizations
(runtime space) for sol3, since I'd like to use a lot less space if
possible. Nothing concrete, but I'll certainly ping this thread if any
of that changes.

Sincerely,
ThePhD