REST Stack for Embedded Device

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

REST Stack for Embedded Device

Andrew Starks-2
Hello All,

At work, we are selecting a Lua environment to run on our embedded ARM 7 device. We have limited memory and we want to keep the footprint small, hence Lua in the first place. Our requirements include:
 
* Lua 5.3 preferred because of the need to work with INT64
* TCP-IP Sockets
* HTTP/S, HLS/TLS
* WebSockets
* DNS / mDNS
* REST Plumbing
* Nice async / concurrency behavior
* Integrates well with main loops
* Works on Windows a nice plus
* Capitalist friendly licensing

My current interest is in lua-http with cqueues, but we aren't sure about compatible REST frameworks and which one the cool kids are using. Any recommendations?

Luvit would be a great option, except that it only works with LuaJIT. We're wary of the daunting task of compiling LuaJIT on this platform and we will miss native 64Bit integers. Our team is familiar with Node.js, hence the appeal.

Other lightweight suggestions?

-- Andrew


Reply | Threaded
Open this post in threaded view
|

Re: REST Stack for Embedded Device

Marcus Mason
I'd probably recommend lua-http and cqueues because not only will it run on 5.3 it also has good concurrency abstractions (cqueues.promise is particularly helpful) and the http library has a good interface. I'd avoid luvit; it's not got enough packages or interop with the rest of lua and I find its coroutine based networking libraries (the coro-X set of packages) to be lacklustre. 

On Wed, 1 May 2019, 03:31 Andrew Starks, <[hidden email]> wrote:
Hello All,

At work, we are selecting a Lua environment to run on our embedded ARM 7 device. We have limited memory and we want to keep the footprint small, hence Lua in the first place. Our requirements include:

* Lua 5.3 preferred because of the need to work with INT64
* TCP-IP Sockets
* HTTP/S, HLS/TLS
* WebSockets
* DNS / mDNS
* REST Plumbing
* Nice async / concurrency behavior
* Integrates well with main loops
* Works on Windows a nice plus
* Capitalist friendly licensing

My current interest is in lua-http with cqueues, but we aren't sure about compatible REST frameworks and which one the cool kids are using. Any recommendations?

Luvit would be a great option, except that it only works with LuaJIT. We're wary of the daunting task of compiling LuaJIT on this platform and we will miss native 64Bit integers. Our team is familiar with Node.js, hence the appeal.

Other lightweight suggestions?

-- Andrew


Reply | Threaded
Open this post in threaded view
|

Re: REST Stack for Embedded Device

Russell Haley
In reply to this post by Andrew Starks-2


On Tue, Apr 30, 2019 at 7:31 PM Andrew Starks <[hidden email]> wrote:
Hello All,

At work, we are selecting a Lua environment to run on our embedded ARM 7 device. We have limited memory and we want to keep the footprint small, hence Lua in the first place. Our requirements include:

* Lua 5.3 preferred because of the need to work with INT64
* TCP-IP Sockets
* HTTP/S, HLS/TLS
* WebSockets
* DNS / mDNS
* REST Plumbing
* Nice async / concurrency behavior
* Integrates well with main loops
* Works on Windows a nice plus
* Capitalist friendly licensing

My current interest is in lua-http with cqueues, but we aren't sure about compatible REST frameworks and which one the cool kids are using. Any recommendations?
lua-http runs great on my beaglebone. 
 

Luvit would be a great option, except that it only works with LuaJIT. We're wary of the daunting task of compiling LuaJIT on this platform and we will miss native 64Bit integers. Our team is familiar with Node.js, hence the appeal.

Other lightweight suggestions?

-- Andrew


Reply | Threaded
Open this post in threaded view
|

Re: REST Stack for Embedded Device

Andrew Starks-2
In reply to this post by Marcus Mason

[topping your top post, just to be social]

 

I’m wondering about REST frameworks. My reading is that lua-http doesn’t really have url routing features and the kinds of API building tools you find in Node.JS, etc.

 

OpenResty/NGINX looks interesting as well, but again, it’s LuaJIT.

 

Nobody is going to load LuaRocks in our use case and so integration with the Luaverse is not a requirement. However, LuaRocks 3.1 does have some interesting features for building containers.

 

Sorry to ramble… :)

 

-Andrew

 

On 4/30/19, 10:16 PM, "[hidden email] on behalf of Marcus Mason" <[hidden email] on behalf of [hidden email]> wrote:

 

I'd probably recommend lua-http and cqueues because not only will it run on 5.3 it also has good concurrency abstractions (cqueues.promise is particularly helpful) and the http library has a good interface. I'd avoid luvit; it's not got enough packages or interop with the rest of lua and I find its coroutine based networking libraries (the coro-X set of packages) to be lacklustre. 

 

On Wed, 1 May 2019, 03:31 Andrew Starks, <[hidden email]> wrote:

Hello All,

At work, we are selecting a Lua environment to run on our embedded ARM 7 device. We have limited memory and we want to keep the footprint small, hence Lua in the first place. Our requirements include:

* Lua 5.3 preferred because of the need to work with INT64
* TCP-IP Sockets
* HTTP/S, HLS/TLS
* WebSockets
* DNS / mDNS
* REST Plumbing
* Nice async / concurrency behavior
* Integrates well with main loops
* Works on Windows a nice plus
* Capitalist friendly licensing

My current interest is in lua-http with cqueues, but we aren't sure about compatible REST frameworks and which one the cool kids are using. Any recommendations?

Luvit would be a great option, except that it only works with LuaJIT. We're wary of the daunting task of compiling LuaJIT on this platform and we will miss native 64Bit integers. Our team is familiar with Node.js, hence the appeal.

Other lightweight suggestions?

-- Andrew

Reply | Threaded
Open this post in threaded view
|

Re: REST Stack for Embedded Device

Benoit Giannangeli
To be honest it’d be pretty easy to write one on top of lua-http. And if you want to keep control over your program’s memory/cpu cost, it’s seems natural to do so.

On Wed 1 May 2019 at 06:22, Andrew Starks <[hidden email]> wrote:

[topping your top post, just to be social]

 

I’m wondering about REST frameworks. My reading is that lua-http doesn’t really have url routing features and the kinds of API building tools you find in Node.JS, etc.

 

OpenResty/NGINX looks interesting as well, but again, it’s LuaJIT.

 

Nobody is going to load LuaRocks in our use case and so integration with the Luaverse is not a requirement. However, LuaRocks 3.1 does have some interesting features for building containers.

 

Sorry to ramble… :)

 

-Andrew

 

On 4/30/19, 10:16 PM, "[hidden email] on behalf of Marcus Mason" <[hidden email] on behalf of [hidden email]> wrote:

 

I'd probably recommend lua-http and cqueues because not only will it run on 5.3 it also has good concurrency abstractions (cqueues.promise is particularly helpful) and the http library has a good interface. I'd avoid luvit; it's not got enough packages or interop with the rest of lua and I find its coroutine based networking libraries (the coro-X set of packages) to be lacklustre. 

 

On Wed, 1 May 2019, 03:31 Andrew Starks, <[hidden email]> wrote:

Hello All,

At work, we are selecting a Lua environment to run on our embedded ARM 7 device. We have limited memory and we want to keep the footprint small, hence Lua in the first place. Our requirements include:

* Lua 5.3 preferred because of the need to work with INT64
* TCP-IP Sockets
* HTTP/S, HLS/TLS
* WebSockets
* DNS / mDNS
* REST Plumbing
* Nice async / concurrency behavior
* Integrates well with main loops
* Works on Windows a nice plus
* Capitalist friendly licensing

My current interest is in lua-http with cqueues, but we aren't sure about compatible REST frameworks and which one the cool kids are using. Any recommendations?

Luvit would be a great option, except that it only works with LuaJIT. We're wary of the daunting task of compiling LuaJIT on this platform and we will miss native 64Bit integers. Our team is familiar with Node.js, hence the appeal.

Other lightweight suggestions?

-- Andrew

--
Benoit Giannangeli
 
Reply | Threaded
Open this post in threaded view
|

Re: REST Stack for Embedded Device

Russell Haley
In reply to this post by Russell Haley


On Tue, Apr 30, 2019 at 8:54 PM Russell Haley <[hidden email]> wrote:


On Tue, Apr 30, 2019 at 7:31 PM Andrew Starks <[hidden email]> wrote:
Hello All,

At work, we are selecting a Lua environment to run on our embedded ARM 7 device. We have limited memory and we want to keep the footprint small, hence Lua in the first place. Our requirements include:

* Lua 5.3 preferred because of the need to work with INT64
* TCP-IP Sockets
* HTTP/S, HLS/TLS
* WebSockets
* DNS / mDNS
* REST Plumbing
* Nice async / concurrency behavior
* Integrates well with main loops
* Works on Windows a nice plus
* Capitalist friendly licensing

My current interest is in lua-http with cqueues, but we aren't sure about compatible REST frameworks and which one the cool kids are using. Any recommendations?
lua-http runs great on my beaglebone. 
Sorry, I was going to write something and got jumped on by my kids. 

Lua-http runs great on my beaglebone, which is a Cortex A-8 (Arm v7a). The memory usage is a little high IMHO, coming in around 7 MB due to a rather large LPEG library. I wouldn't think writing a URL handler for a REST API would be all that difficult seeing as lua-http has many of the other bells and whistles you're looking for. Cqueues provides DNS as well: see section 3.2.7 in the cqueues manual here: https://www.25thandclement.com/~william/projects/cqueues.pdf
 
I run everything on FreeBSD, making the whole stack "capitalist friendly".

Here is my websocket endpoints prototypes: 

I also created an application to run minecraft servers in lpty and serve out the console output via a websocket (albeit on amd64). So, effectively it's a lua/cqueues application that runs lua-http within it. 

I would say the difficulty for me using cqueues is that everything runs in a loop so it's not a programming model that I was terribly familiar with (lua-http abstracts that away for the most part). I haven't touched my minecraft runner for some time, but I was having problems with exceptions causing cqueues to jump out of the loop. I'm sure a more skilled programmer that has time to work on these things would be able to deal with that (these are all hobby projects that I do in the 1 1/2 hours I have after putting kids to bed). 

FWIW,
Russ
 

Luvit would be a great option, except that it only works with LuaJIT. We're wary of the daunting task of compiling LuaJIT on this platform and we will miss native 64Bit integers. Our team is familiar with Node.js, hence the appeal.

Other lightweight suggestions?

-- Andrew


Reply | Threaded
Open this post in threaded view
|

Re: REST Stack for Embedded Device

Ryan Ford-2
In reply to this post by Andrew Starks-2
You can use Lapis with a cqueues backend. I have 2 sites running with that set up, works great.

On Wed, 1 May 2019 02:30:57 +0000
Andrew Starks <[hidden email]> wrote:

> Hello All,
>
> At work, we are selecting a Lua environment to run on our embedded ARM 7 device. We have limited memory and we want to keep the footprint small, hence Lua in the first place. Our requirements include:
>  
> * Lua 5.3 preferred because of the need to work with INT64
> * TCP-IP Sockets
> * HTTP/S, HLS/TLS
> * WebSockets
> * DNS / mDNS
> * REST Plumbing
> * Nice async / concurrency behavior
> * Integrates well with main loops
> * Works on Windows a nice plus
> * Capitalist friendly licensing
>
> My current interest is in lua-http with cqueues, but we aren't sure about compatible REST frameworks and which one the cool kids are using. Any recommendations?
>
> Luvit would be a great option, except that it only works with LuaJIT. We're wary of the daunting task of compiling LuaJIT on this platform and we will miss native 64Bit integers. Our team is familiar with Node.js, hence the appeal.
>
> Other lightweight suggestions?
>
> -- Andrew
>
>

Reply | Threaded
Open this post in threaded view
|

Re: REST Stack for Embedded Device

Ryan Ford-2
If that's too big you could use something like https://github.com/APItools/router.lua to just get some REST/CRUD stuff going and use bare cqueues.

On Wed, 1 May 2019 13:04:12 +0800
Ryan Ford <[hidden email]> wrote:

> You can use Lapis with a cqueues backend. I have 2 sites running with that set up, works great.
>
> On Wed, 1 May 2019 02:30:57 +0000
> Andrew Starks <[hidden email]> wrote:
>
> > Hello All,
> >
> > At work, we are selecting a Lua environment to run on our embedded ARM 7 device. We have limited memory and we want to keep the footprint small, hence Lua in the first place. Our requirements include:
> >  
> > * Lua 5.3 preferred because of the need to work with INT64
> > * TCP-IP Sockets
> > * HTTP/S, HLS/TLS
> > * WebSockets
> > * DNS / mDNS
> > * REST Plumbing
> > * Nice async / concurrency behavior
> > * Integrates well with main loops
> > * Works on Windows a nice plus
> > * Capitalist friendly licensing
> >
> > My current interest is in lua-http with cqueues, but we aren't sure about compatible REST frameworks and which one the cool kids are using. Any recommendations?
> >
> > Luvit would be a great option, except that it only works with LuaJIT. We're wary of the daunting task of compiling LuaJIT on this platform and we will miss native 64Bit integers. Our team is familiar with Node.js, hence the appeal.
> >
> > Other lightweight suggestions?
> >
> > -- Andrew
> >
> >
>

Reply | Threaded
Open this post in threaded view
|

Re: REST Stack for Embedded Device

Russell Haley
In reply to this post by Andrew Starks-2


On Tue, Apr 30, 2019 at 7:31 PM Andrew Starks <[hidden email]> wrote:
Hello All,

At work, we are selecting a Lua environment to run on our embedded ARM 7 device. We have limited memory and we want to keep the footprint small, hence Lua in the first place. Our requirements include:

* Lua 5.3 preferred because of the need to work with INT64
* TCP-IP Sockets
* HTTP/S, HLS/TLS
* WebSockets
* DNS / mDNS
* REST Plumbing
* Nice async / concurrency behavior
* Integrates well with main loops
* Works on Windows a nice plus
* Capitalist friendly licensing

My current interest is in lua-http with cqueues, but we aren't sure about compatible REST frameworks and which one the cool kids are using. Any recommendations?

Luvit would be a great option, except that it only works with LuaJIT. We're wary of the daunting task of compiling LuaJIT on this platform and we will miss native 64Bit integers. Our team is familiar with Node.js, hence the appeal.

Here's another question that hasn't been asked or answered: Are you going to need an authentication scheme? Node.js has passport that has modules for every authentication model you can think of. lua-http doesn't have anything built in that I'm aware of.  

Other lightweight suggestions?

-- Andrew


Reply | Threaded
Open this post in threaded view
|

Re: REST Stack for Embedded Device

Daurnimator
In reply to this post by Russell Haley
On Wed, 1 May 2019 at 14:49, Russell Haley <[hidden email]> wrote:
> The memory usage is a little high IMHO, coming in around 7 MB due to a rather large LPEG library.

FWIW back in september I did a refactor of lpeg_patterns to reduce memory usage.
There hasn't been a release since.

Reply | Threaded
Open this post in threaded view
|

Re: REST Stack for Embedded Device

Andrew Starks-2
In reply to this post by Russell Haley


Andrew Starks

From: [hidden email] on behalf of Russell Haley <[hidden email]>
Sent: Wednesday, May 1, 2019 00:25
To: Lua mailing list
Subject: Re: REST Stack for Embedded Device
 


On Tue, Apr 30, 2019 at 7:31 PM Andrew Starks <[hidden email]> wrote:
Hello All,

At work, we are selecting a Lua environment to run on our embedded ARM 7 device. We have limited memory and we want to keep the footprint small, hence Lua in the first place. Our requirements include:

* Lua 5.3 preferred because of the need to work with INT64
* TCP-IP Sockets
* HTTP/S, HLS/TLS
* WebSockets
* DNS / mDNS
* REST Plumbing
* Nice async / concurrency behavior
* Integrates well with main loops
* Works on Windows a nice plus
* Capitalist friendly licensing

My current interest is in lua-http with cqueues, but we aren't sure about compatible REST frameworks and which one the cool kids are using. Any recommendations?

Luvit would be a great option, except that it only works with LuaJIT. We're wary of the daunting task of compiling LuaJIT on this platform and we will miss native 64Bit integers. Our team is familiar with Node.js, hence the appeal.

Here's another question that hasn't been asked or answered: Are you going to need an authentication scheme? Node.js has passport that has modules for every authentication model you can think of. lua-http doesn't have anything built in that I'm aware of. 

———

AS: yes. I do need authentication. OAuth is preferred, but there is fuzziness on version. A well curated variety of options is a plus.  

- Andrew 

Reply | Threaded
Open this post in threaded view
|

Re: REST Stack for Embedded Device

François Perrad
In reply to this post by Andrew Starks-2


Le mer. 1 mai 2019 à 06:22, Andrew Starks <[hidden email]> a écrit :

[topping your top post, just to be social]

 

I’m wondering about REST frameworks. My reading is that lua-http doesn’t really have url routing features and the kinds of API building tools you find in Node.JS, etc.

 


lua-Rotas (https://fperrad.frama.io/lua-Rotas/rotas/) could help you as web server router.

François
 

OpenResty/NGINX looks interesting as well, but again, it’s LuaJIT.

 

Nobody is going to load LuaRocks in our use case and so integration with the Luaverse is not a requirement. However, LuaRocks 3.1 does have some interesting features for building containers.

 

Sorry to ramble… :)

 

-Andrew

 

Reply | Threaded
Open this post in threaded view
|

Re: REST Stack for Embedded Device

云风 Cloud Wu
In reply to this post by Andrew Starks-2


在 2019年5月1日,10:30,Andrew Starks <[hidden email]> 写道:

Hello All,

At work, we are selecting a Lua environment to run on our embedded ARM 7 device. We have limited memory and we want to keep the footprint small, hence Lua in the first place. Our requirements include:

* Lua 5.3 preferred because of the need to work with INT64
* TCP-IP Sockets
* HTTP/S, HLS/TLS
* WebSockets
* DNS / mDNS
* REST Plumbing
* Nice async / concurrency behavior
* Integrates well with main loops
* Works on Windows a nice plus
* Capitalist friendly licensing

My current interest is in lua-http with cqueues, but we aren't sure about compatible REST frameworks and which one the cool kids are using. Any recommendations?

Luvit would be a great option, except that it only works with LuaJIT. We're wary of the daunting task of compiling LuaJIT on this platform and we will miss native 64Bit integers. Our team is familiar with Node.js, hence the appeal.

Other lightweight suggestions?

-- Andrew



Maybe my project ‘skynet’ is a choice, you can have a look at https://github.com/cloudwu/skynet

It’s similar with cqueues.

It’s widely used in many online game severs, and someone reported that they used it in some embedded environments. I maintains it for eight years, I think it is robust enough, but there is no English documents yet because my English is not good. 



Reply | Threaded
Open this post in threaded view
|

Re: REST Stack for Embedded Device

Aaron B.
In reply to this post by Andrew Starks-2
On Wed, 1 May 2019 02:30:57 +0000
Andrew Starks <[hidden email]> wrote:

>
> My current interest is in lua-http with cqueues, but we aren't sure about compatible REST frameworks and which one the cool kids are using. Any recommendations?
>

Perihelion (https://zadzmo.org/code/perihelion) makes REST API's quite
easy to build. At this point, it's been battle tested on Xavante and
uWSGI.

Getting it running under lua-http/cqueues is on the todo list; I don't
expect that to be hard.

--
Aaron B. <[hidden email]>

Reply | Threaded
Open this post in threaded view
|

Re: REST Stack for Embedded Device

Andrew Starks-2
In reply to this post by 云风 Cloud Wu


On Thu, May 2, 2019 at 10:30 AM 云风 <[hidden email]> wrote:


在 2019年5月1日,10:30,Andrew Starks <[hidden email]> 写道:

Hello All,

At work, we are selecting a Lua environment to run on our embedded ARM 7 device. We have limited memory and we want to keep the footprint small, hence Lua in the first place. Our requirements include:

* Lua 5.3 preferred because of the need to work with INT64
* TCP-IP Sockets
* HTTP/S, HLS/TLS
* WebSockets
* DNS / mDNS
* REST Plumbing
* Nice async / concurrency behavior
* Integrates well with main loops
* Works on Windows a nice plus
* Capitalist friendly licensing

My current interest is in lua-http with cqueues, but we aren't sure about compatible REST frameworks and which one the cool kids are using. Any recommendations?

Luvit would be a great option, except that it only works with LuaJIT. We're wary of the daunting task of compiling LuaJIT on this platform and we will miss native 64Bit integers. Our team is familiar with Node.js, hence the appeal.

Other lightweight suggestions?

-- Andrew



Maybe my project ‘skynet’ is a choice, you can have a look at https://github.com/cloudwu/skynet

It’s similar with cqueues.

It’s widely used in many online game severs, and someone reported that they used it in some embedded environments. I maintains it for eight years, I think it is robust enough, but there is no English documents yet because my English is not good. 


This is very interesting. Your use case is similar to mine, but I don't know Chinese. It's hard to pay for someone to translate when we are only evaluating.

If you could point me to a case study or examples of your framework in use, I would appreciate that very much.

--
Andrew Starks
Reply | Threaded
Open this post in threaded view
|

Re: REST Stack for Embedded Device

云风 Cloud Wu


在 2019年5月3日,11:21,Andrew Starks <[hidden email]> 写道:



On Thu, May 2, 2019 at 10:30 AM 云风 <[hidden email]> wrote:


在 2019年5月1日,10:30,Andrew Starks <[hidden email]> 写道:

Hello All,

At work, we are selecting a Lua environment to run on our embedded ARM 7 device. We have limited memory and we want to keep the footprint small, hence Lua in the first place. Our requirements include:

* Lua 5.3 preferred because of the need to work with INT64
* TCP-IP Sockets
* HTTP/S, HLS/TLS
* WebSockets
* DNS / mDNS
* REST Plumbing
* Nice async / concurrency behavior
* Integrates well with main loops
* Works on Windows a nice plus
* Capitalist friendly licensing

My current interest is in lua-http with cqueues, but we aren't sure about compatible REST frameworks and which one the cool kids are using. Any recommendations?

Luvit would be a great option, except that it only works with LuaJIT. We're wary of the daunting task of compiling LuaJIT on this platform and we will miss native 64Bit integers. Our team is familiar with Node.js, hence the appeal.

Other lightweight suggestions?

-- Andrew



Maybe my project ‘skynet’ is a choice, you can have a look at https://github.com/cloudwu/skynet

It’s similar with cqueues.

It’s widely used in many online game severs, and someone reported that they used it in some embedded environments. I maintains it for eight years, I think it is robust enough, but there is no English documents yet because my English is not good. 


This is very interesting. Your use case is similar to mine, but I don't know Chinese. It's hard to pay for someone to translate when we are only evaluating.

If you could point me to a case study or examples of your framework in use, I would appreciate that very much.

--
Andrew Starks

Building skynet is very simple, just clone it, and type ‘make linux’. 

You can run 

./skynet examples/config

And then run some test or examples script in the console, for example type ‘simpleweb’ to run the script in examples/simpleweb.lua


I think  it’s easy to understand how the skynet works as a web server.

The core of skynet is lightweight , less then 10k lines of C code , it’s only an actor framework like Erlang and with async  tcp/udp support. There are http(s) modules written by pure lua, and some db drivers such as redis/mongo/mysql base on this async tcp api.

Tou can find some test script at https://github.com/cloudwu/skynet/tree/master/test

You can send me a direct email if you need more help.