Lua use case questions

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Lua use case questions

Kevin Wang
Hi everyone,

I am just getting into Lua to research it as a possible language for a
grant we are working on, and was wondering if anyone could answer some
questions for a beginner :)

We write software and hardware for neuroscience experiments, and are
developing a new tool. Our ultimate goal in a nutshell is to have users
be able to write and run scripts on their computers, but only as
simulation and without low latency. The idea is that once they are
satisfied with fine tuning their control script and parameters, they can
then "upload" it to our device (a soft-core processor on an FPGA), in
which case our backend can execute their control script with direct
access to the data (going through the FPGA) for extremely low latency.

So my questions are:

1. Is Lua syntax easy to learn, especially for users that primarily code
analysis scripts in Matlab or Python? (Even if it is not, if Lua turns
out to meet our tech requirements, they'll just have to learn Lua
anyways. Just a good thing for me to keep in mind)

2. Is this a fair use case for Lua? I know that Lua is small enough to
use on embedded devices, what about an FPGA?

3. Does anyone here have a similar use case that I could talk to, or see
any similar examples for?

4. If not Lua, any other recommendations?

Many many thanks,

Kevin


Reply | Threaded
Open this post in threaded view
|

Re: Lua use case questions

Russell Haley


On Wed, May 30, 2018 at 10:06 AM, Kevin Wang <[hidden email]> wrote:
Hi everyone,

I am just getting into Lua to research it as a possible language for a grant we are working on, and was wondering if anyone could answer some questions for a beginner :)

We write software and hardware for neuroscience experiments, and are developing a new tool. Our ultimate goal in a nutshell is to have users be able to write and run scripts on their computers, but only as simulation and without low latency. The idea is that once they are satisfied with fine tuning their control script and parameters, they can then "upload" it to our device (a soft-core processor on an FPGA), in which case our backend can execute their control script with direct access to the data (going through the FPGA) for extremely low latency.

So my questions are:

1. Is Lua syntax easy to learn, especially for users that primarily code analysis scripts in Matlab or Python? (Even if it is not, if Lua turns out to meet our tech requirements, they'll just have to learn Lua anyways. Just a good thing for me to keep in mind)
 
IHMO Lua is very easy to learn because the language is so small. That said, there are things about Lua that are not friendly to non-technical people (meta-tables, functional programming, upvalues) but very powerful for intermediate to advanced users. It also has some quirks, but overall is a very simple language. One thing to note is that Lua string manipulation and pattern matching  is *very* different than what most people expect. Once a person is familiarized with the Lua string patterns, it's hard to go back to indexes and regex. However, string manipulation was a hefty speed bump for me and it could be a considerable learning curve for some.

In terms of learnability and performance: Lua is used extensively in games for user extensions. As an example, Roblox games are written entirely in Lua by the Roblox users.  I'm not sure if they are using Lua "Proper" or LuaJIT though (the language was forked some years back).

Where things tend to get difficult in Lua is the lack of a standard library. That's a good thing in some senses (no bloat), but it means you're going to need to provide an API for your users by importing or writing Lua Modules. You might be able to get away with just importing something like Penlight, (Moses would be good if you're making calculations), but Lua the language gives you very little for free (compared to say Python on Java). 

Another thing will be squashing down the Lua scripts and getting them onto the embedded system and loading them. The tool called squish seems to be a good one, though I had no success with it in my 5 minute attempts on Windows. 


2. Is this a fair use case for Lua? I know that Lua is small enough to use on embedded devices, what about an FPGA?
- eLua is designed for microcontrollers and what-have you. I haven't used eLua but have a lot of interest in it: http://www.eluaproject.net/overview 
- I've been coming up to speed on RTEMS and recently got a hello world working with that. RTEMS can be run on an FPGA configured as a LEON2 processor - but I don't have an FPGA. I'm using the ERC32 simulator in a FreeBSD VM.


3. Does anyone here have a similar use case that I could talk to, or see any similar examples for?
Your use case is what I am *trying* to do: Allow people to write scripts on Windows or Linux and then run it on something embedded.  On the user scripting side, I have a WinLua project that is supposed to help people start programming in Lua on Windows. ZeroBrane Studio IDE would also be a good platform for your "user scripting interface". 

  

4. If not Lua, any other recommendations?

I personally found/find the Lua C API difficult so I've started including a header only C++ wrapper called Sol2 in my WinLua project. If you support C++ 17 it might be worth looking at. 

The term you are looking for is "Glue Language". TCL is an example, but good luck on the syntax. A person named Peter MacDonald (created the first Linux Distro) as created an embeddable javascript interpreter called jsish (JavaScript Interpreter Shell). jsish.org. He has taken a different approach and included everything he thinks you'll need as an embedded developer. Jsi comes with a socket library, websockets, sqlite integration and a built in debugger (command line or web interface). Those tools will cost you in terms of size though.

Are you looking for Machine learning? Check out https://github.com/dibyendumajumdar/ravi-torch7 (though I think it's broken right now).


Many many thanks,
Good Luck, please keep us informed of you're progress, I'd love to hear more!
Russ 

Kevin



Reply | Threaded
Open this post in threaded view
|

Re: Lua use case questions

Frank Kastenholz-2


On May 30, 2018, at 4:45 PM, Russell Haley <[hidden email]> wrote:



On Wed, May 30, 2018 at 10:06 AM, Kevin Wang <[hidden email]> wrote:
Hi everyone,

I am just getting into Lua to research it as a possible language for a grant we are working on, and was wondering if anyone could answer some questions for a beginner :)

We write software and hardware for neuroscience experiments, and are developing a new tool. Our ultimate goal in a nutshell is to have users be able to write and run scripts on their computers, but only as simulation and without low latency. The idea is that once they are satisfied with fine tuning their control script and parameters, they can then "upload" it to our device (a soft-core processor on an FPGA), in which case our backend can execute their control script with direct access to the data (going through the FPGA) for extremely low latency.

So my questions are:

1. Is Lua syntax easy to learn, especially for users that primarily code analysis scripts in Matlab or Python? (Even if it is not, if Lua turns out to meet our tech requirements, they'll just have to learn Lua anyways. Just a good thing for me to keep in mind)

On a previous project I had a similar issue.
The application developers fiercely resisted our choice of using Lua. As best I could figure out, this was all “I don’t want to learn a language that won’t help me get my next job”.

You might run in to this form of resistance.

We had two real counter arguments
1) licensing meant that Python, Java, (the app developers favorites) etc we’re not options and
2) PiL has “all you need to know about Lua” and it’s like 1/4 the size of the equivalent Python, etc, books from O’Reilly.  In other words, “Lua is easy”.

Our app developers became reasonably proficient in Lua in about a week, learning the library & environment my team was developing took another week or so

IHMO Lua is very easy to learn because the language is so small. That said, there are things about Lua that are not friendly to non-technical people (meta-tables, functional programming, upvalues) but very powerful for intermediate to advanced users.

Agreed
With the addition that these advanced things are not mandatory for core proficiency in writing basic applications


In terms of learnability and performance: Lua is used extensively in games ...

I would also add that app developers might be assuming that the compiler does a lot of optimization, etc. PUC-Rio Lua does not do any optimization.
This was an issue in my old project.  The app team said that the Lua version of their code ran 50-500x slower than C.
I spent about 2 days getting that to 5-10x slower. I just did some basic optimizations that we all learned when we started programming (back in the dark ages).  
(The app developers really didn’t like me or Lua after that ... but that’s another, nontechnical, story :-)


Where things tend to get difficult in Lua is the lack of a standard library.

Agreed

2. Is this a fair use case for Lua? I know that Lua is small enough to use on embedded devices, what about an FPGA?

Yes - ish
If Lua will be running on some standard cpu with an OS of some sort, etc, and you’d have things that look like function calls that actually send data into the FPGA to do its magic.

The project I was on did sort of the same thing. It worked well ... but you need to put time into figuring out the programming model for the lua apps and the interfaces that the model requires.  In addition you need to put a lot of time into designing the api between the Lua apps and the special functions embedded in the FPGA.

4. If not Lua, any other recommendations?

For my project we looked at Java, JavaScript, and Python. All were rehjectwd because of size/performance issues and/or the license being unacceptable

Frank Kastenholz

Reply | Threaded
Open this post in threaded view
|

Re: Lua use case questions

Kevin Wang

Thanks for your replies.

It sounds like Lua is pretty easy to learn if people need to. The users probably wouldn't need to use metatables anyways. What we will have to do is write an API for them to access the data in the FPGA, set callback functions, and execute specific commands on the device.

I would also add that app developers might be assuming that the compiler does a lot of optimization, etc. PUC-Rio Lua does not do any optimization.
This was an issue in my old project.  The app team said that the Lua version of their code ran 50-500x slower than C.
I spent about 2 days getting that to 5-10x slower. I just did some basic optimizations that we all learned when we started programming (back in the dark ages).  

Did you make the optimizations in your code or in the Lua interpreter?

Why not use LuaJIT? What do people think of LuaJIT on here? Are there any cons to using LuaJIT, besides the extra space required?

Kevin


On 05/31/2018 05:50 AM, Frank Kastenholz wrote:


On May 30, 2018, at 4:45 PM, Russell Haley <[hidden email]> wrote:



On Wed, May 30, 2018 at 10:06 AM, Kevin Wang <[hidden email]> wrote:
Hi everyone,

I am just getting into Lua to research it as a possible language for a grant we are working on, and was wondering if anyone could answer some questions for a beginner :)

We write software and hardware for neuroscience experiments, and are developing a new tool. Our ultimate goal in a nutshell is to have users be able to write and run scripts on their computers, but only as simulation and without low latency. The idea is that once they are satisfied with fine tuning their control script and parameters, they can then "upload" it to our device (a soft-core processor on an FPGA), in which case our backend can execute their control script with direct access to the data (going through the FPGA) for extremely low latency.

So my questions are:

1. Is Lua syntax easy to learn, especially for users that primarily code analysis scripts in Matlab or Python? (Even if it is not, if Lua turns out to meet our tech requirements, they'll just have to learn Lua anyways. Just a good thing for me to keep in mind)

On a previous project I had a similar issue.
The application developers fiercely resisted our choice of using Lua. As best I could figure out, this was all “I don’t want to learn a language that won’t help me get my next job”.

You might run in to this form of resistance.

We had two real counter arguments
1) licensing meant that Python, Java, (the app developers favorites) etc we’re not options and
2) PiL has “all you need to know about Lua” and it’s like 1/4 the size of the equivalent Python, etc, books from O’Reilly.  In other words, “Lua is easy”.

Our app developers became reasonably proficient in Lua in about a week, learning the library & environment my team was developing took another week or so

IHMO Lua is very easy to learn because the language is so small. That said, there are things about Lua that are not friendly to non-technical people (meta-tables, functional programming, upvalues) but very powerful for intermediate to advanced users.

Agreed
With the addition that these advanced things are not mandatory for core proficiency in writing basic applications


In terms of learnability and performance: Lua is used extensively in games ...

I would also add that app developers might be assuming that the compiler does a lot of optimization, etc. PUC-Rio Lua does not do any optimization.
This was an issue in my old project.  The app team said that the Lua version of their code ran 50-500x slower than C.
I spent about 2 days getting that to 5-10x slower. I just did some basic optimizations that we all learned when we started programming (back in the dark ages).  
(The app developers really didn’t like me or Lua after that ... but that’s another, nontechnical, story :-)


Where things tend to get difficult in Lua is the lack of a standard library.

Agreed

2. Is this a fair use case for Lua? I know that Lua is small enough to use on embedded devices, what about an FPGA?

Yes - ish
If Lua will be running on some standard cpu with an OS of some sort, etc, and you’d have things that look like function calls that actually send data into the FPGA to do its magic.

The project I was on did sort of the same thing. It worked well ... but you need to put time into figuring out the programming model for the lua apps and the interfaces that the model requires.  In addition you need to put a lot of time into designing the api between the Lua apps and the special functions embedded in the FPGA.

4. If not Lua, any other recommendations?

For my project we looked at Java, JavaScript, and Python. All were rehjectwd because of size/performance issues and/or the license being unacceptable

Frank Kastenholz


Reply | Threaded
Open this post in threaded view
|

Re: Lua use case questions

Sean Conner
It was thus said that the Great Kevin Wang once stated:
>
> Why not use LuaJIT? What do people think of LuaJIT on here? Are there
> any cons to using LuaJIT, besides the extra space required?

  LuaJIT will only JIT code for a limited number of architectures (x86 and
ARM).  Second, LuaJIT is still Lua 5.1 with some additions that make it
close to Lua 5.2 without being Lua 5.2; Lua 5.3 is the current version of
standard Lua.  The major differences are how modules are coded (drastically
changed from Lua 5.1 to 5.2) and 64-bit integer support (Lua 5.3).

  -spc


Reply | Threaded
Open this post in threaded view
|

Re: Lua use case questions

Russell Haley


On Thu, May 31, 2018 at 10:39 AM, Sean Conner <[hidden email]> wrote:
It was thus said that the Great Kevin Wang once stated:
>
> Why not use LuaJIT? What do people think of LuaJIT on here? Are there
> any cons to using LuaJIT, besides the extra space required?

  LuaJIT will only JIT code for a limited number of architectures (x86 and
ARM).  Second, LuaJIT is still Lua 5.1 with some additions that make it
close to Lua 5.2 without being Lua 5.2; Lua 5.3 is the current version of
standard Lua.  The major differences are how modules are coded (drastically
changed from Lua 5.1 to 5.2) and 64-bit integer support (Lua 5.3).

LuaJIT doesn't differentiate between int and float either. That was just *TOO MUCH* for Mike Pall. 

 

  -spc



Reply | Threaded
Open this post in threaded view
|

Re: Lua use case questions

Frank Kastenholz-2
In reply to this post by Kevin Wang


On May 31, 2018, at 1:18 PM, Kevin Wang <[hidden email]> wrote:
I would also add that app developers might be assuming that the compiler does a lot of optimization, etc. PUC-Rio Lua does not do any optimization.
This was an issue in my old project.  The app team said that the Lua version of their code ran 50-500x slower than C.
I spent about 2 days getting that to 5-10x slower. I just did some basic optimizations that we all learned when we started programming (back in the dark ages).  

Did you make the optimizations in your code or in the Lua interpreter?

I optimized the application, not Lua.
For example
- we changed global to locals.
- the original code had an if statement in a very large loop in a function. The if determined what processing was done & it didn’t change for any given invocation of the function:

Function x(flag)
   Local I
   For I=1,10000000 do
         If Flag then
             Process1
         Else
             Process2
          End
    End
End

We took the if out of the loop:

Function x(flag)
   Local I
    If flag then
       For I=1,10000000 do
             Process1
        End
     Else
         For I=1,10000000
             Process2
          End
    End
End

Modern optimizing compilers do this automatically for, eg, C. 

Why not use LuaJIT? What do people think of LuaJIT on here? Are there any cons to using LuaJIT, besides the extra space required?

LuaJIT wasn’t a viable choice at the time of this project. 
It wasn’t supported and was compatible with an earlier version of Lua.
It makes use of highly tuned fragments of pre-made assembler. Also, IIRC, it would dynamically generate some assembly code. Neither of these were possible in our target environment.

These all were specific to our environment and customer — your situation may well be different

Frank Kastenholz

Reply | Threaded
Open this post in threaded view
|

Re: Lua use case questions

Thijs Schreijer
In reply to this post by Sean Conner


> On 31 May 2018, at 19:39, Sean Conner <[hidden email]> wrote:
>
> It was thus said that the Great Kevin Wang once stated:
>>
>> Why not use LuaJIT? What do people think of LuaJIT on here? Are there
>> any cons to using LuaJIT, besides the extra space required?
>
>  LuaJIT will only JIT code for a limited number of architectures (x86 and
> ARM).  Second, LuaJIT is still Lua 5.1 with some additions that make it
> close to Lua 5.2 without being Lua 5.2; Lua 5.3 is the current version of
> standard Lua.  The major differences are how modules are coded (drastically
> changed from Lua 5.1 to 5.2) and 64-bit integer support (Lua 5.3).
>
>  -spc
>
>

LuaJIT is very fast, so that would be a big bonus I guess. Downside it that if you really want to make the most of it, you have to write code in a way that allows LuaJIT to do its magic. So my guess is that your end users will probably learn and write "regular Lua” and that the performance benefits of using LuaJIT will be less. (Still faster, but less faster than possible… how much less…. depends…)

Thijs
Reply | Threaded
Open this post in threaded view
|

Re: Lua use case questions

Bulat Ziganshin
In reply to this post by Sean Conner
Hello Sean,

Thursday, May 31, 2018, 8:39:34 PM, you wrote:

> standard Lua.  The major differences are how modules are coded (drastically
> changed from Lua 5.1 to 5.2) and 64-bit integer support (Lua 5.3).

isn't that changed in 5.1? afair, 5.2 main change is _ENV stuff


--
Best regards,
 Bulat                            mailto:[hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Lua use case questions

Jonathan Goble
On Sat, Jun 2, 2018, 5:00 PM Bulat Ziganshin <[hidden email]> wrote:
Hello Sean,

Thursday, May 31, 2018, 8:39:34 PM, you wrote:

> standard Lua.  The major differences are how modules are coded (drastically
> changed from Lua 5.1 to 5.2) and 64-bit integer support (Lua 5.3).

isn't that changed in 5.1? afair, 5.2 main change is _ENV stuff

...which directly impacted the coding of modules by eliminating setfenv() (sp?) and module(), the latter of which was used in almost all pure Lua 5.1 modules. There were also significant breaking changes to the C functions for setting up extension modules, which now force (if compatibility with both 5.1 and 5.2/5.3 is desired) the use of #if/#else/#endif directives to take a different code path for 5.1. 
Reply | Threaded
Open this post in threaded view
|

Re: Lua use case questions

Dirk Laurie-2
2018-06-03 0:26 GMT+02:00 Jonathan Goble <[hidden email]>:

> ...which directly impacted the coding of modules by eliminating setfenv()
> (sp?) and module(), the latter of which was used in almost all pure Lua 5.1
> modules. There were also significant breaking changes to the C functions for
> setting up extension modules, which now force (if compatibility with both
> 5.1 and 5.2/5.3 is desired) the use of #if/#else/#endif directives to take a
> different code path for 5.1.

You need such directives anyway even between 5.2 and 5.3.

As someone who came to Lua just as 5.2 beta was being released, I am
not very literate in 5.1.

In hindsight, since Lua 5.1 ruled for six years, its successor should
have been named Lua 6.0. Then backwards compatibility would
psychologically seem to be an unreasonable expectation.

Reply | Threaded
Open this post in threaded view
|

Re: Lua use case questions

Soni "They/Them" L.
In reply to this post by Jonathan Goble


On 2018-06-02 07:26 PM, Jonathan Goble wrote:

> On Sat, Jun 2, 2018, 5:00 PM Bulat Ziganshin
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     Hello Sean,
>
>     Thursday, May 31, 2018, 8:39:34 PM, you wrote:
>
>     > standard Lua.  The major differences are how modules are coded
>     (drastically
>     > changed from Lua 5.1 to 5.2) and 64-bit integer support (Lua 5.3).
>
>     isn't that changed in 5.1? afair, 5.2 main change is _ENV stuff
>
>
> ...which directly impacted the coding of modules by eliminating
> setfenv() (sp?) and module(), the latter of which was used in almost
> all pure Lua 5.1 modules. There were also significant breaking changes
> to the C functions for setting up extension modules, which now force
> (if compatibility with both 5.1 and 5.2/5.3 is desired) the use of
> #if/#else/#endif directives to take a different code path for 5.1.
>

Actually, can't you just build the module tables by hand, as you would
in Lua 5.1? You don't get upvalues (because C API upvalues are still a
pain), but...