[ANN] Haxe Lua : compile Lua/LuaJIT script from Haxe 3.3

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

[ANN] Haxe Lua : compile Lua/LuaJIT script from Haxe 3.3

Justin Donaldson
Hi folks,

I recently wrote a Lua target for the Haxe language: http://haxe.org/blog/hello-lua

I think it has a lot of promise for building more sophisticated Lua apps (e.g. Nginx/lua + Redis/lua + javascript/html5/ios), while also granting the ability for those apps to integrate with other languages and frameworks.

The target is still considered to be in beta mode, but passes our extensive unit test suite.  I'd really like to get feedback from the Lua folks here on the structure/performance of the Lua generated code, and how I might better provide extern support (e.g. Haxe doesn't support multiple returns, which I know is a major sticking point at present.)

I hope some of you find it to be useful!
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Haxe Lua : compile Lua/LuaJIT script from Haxe 3.3

stepa alimov
Hello, Justin!

> I'd really like to get feedback from the Lua folks here on the structure/performance of the Lua generated code
I've worked a bit with this generator and what I can say:
in-place increment, decrement operations converts into (function() ... end)() this is good, but not always necessary.
Some AST preprocessing could be made to detect if return of this operations are required. What I mean:
var i = 2;
i++;
print(i);

now converts into something like:

local i = 2
(function()
local hx_tmp = i
i = i + 1
return hx_tmp
end)()
print(i)

Also, long chain of table field dereferencing generates on calls, and this is major performance issue IMO.
Consider we have class Foo in package com.app.super.my.package1.package2.package3.Bar
Then, each call to static field of this class will look like

local instance = com.app.super.my.package1.package2.package3.Bar.Foo.new()

as you can see, there are 9 table lookups

For my work project I've changed generator to produce "cached" classes, so class declaration looks like:
com.app.super.my.package1.package2.package3.Bar.Foo = _hx_new();
local
com_app_super_my_package1_package2_package3_Bar_Foo =
com.app.super.my.package1.package2.package3.Bar.Foo

Along with removing stuff needed for reflection (code I converting doesn't use this). I've got about 2 times performance boost.

I'm using lua5.2 as target, but under LuaJIT with and without jit-compilation 2-times boost also persists.

Hope my feedback was helpful and thanks for your great work!

2016-05-29 14:16 GMT+03:00 Justin Donaldson <[hidden email]>:
Hi folks,

I recently wrote a Lua target for the Haxe language: http://haxe.org/blog/hello-lua

I think it has a lot of promise for building more sophisticated Lua apps (e.g. Nginx/lua + Redis/lua + javascript/html5/ios), while also granting the ability for those apps to integrate with other languages and frameworks.

The target is still considered to be in beta mode, but passes our extensive unit test suite.  I'd really like to get feedback from the Lua folks here on the structure/performance of the Lua generated code, and how I might better provide extern support (e.g. Haxe doesn't support multiple returns, which I know is a major sticking point at present.)

I hope some of you find it to be useful!



--
С уважением, Алимов Степан Геннадьевич.
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Haxe Lua : compile Lua/LuaJIT script from Haxe 3.3

Justin Donaldson
The cached class concept is great idea.  I know the javascript target uses a similar method, and I'll look into making it a default for lua.  FWIW, you can easily generate "humanized" static method names with the @:export("new_method_name) metadata on the respective field.


On Sun, May 29, 2016 at 2:05 PM, stepa alimov <[hidden email]> wrote:
Hello, Justin!

> I'd really like to get feedback from the Lua folks here on the structure/performance of the Lua generated code
I've worked a bit with this generator and what I can say:
in-place increment, decrement operations converts into (function() ... end)() this is good, but not always necessary.
Some AST preprocessing could be made to detect if return of this operations are required. What I mean:
var i = 2;
i++;
print(i);

now converts into something like:

local i = 2
(function()
local hx_tmp = i
i = i + 1
return hx_tmp
end)()
print(i)

Also, long chain of table field dereferencing generates on calls, and this is major performance issue IMO.
Consider we have class Foo in package com.app.super.my.package1.package2.package3.Bar
Then, each call to static field of this class will look like

local instance = com.app.super.my.package1.package2.package3.Bar.Foo.new()

as you can see, there are 9 table lookups

For my work project I've changed generator to produce "cached" classes, so class declaration looks like:
com.app.super.my.package1.package2.package3.Bar.Foo = _hx_new();
local
com_app_super_my_package1_package2_package3_Bar_Foo =
com.app.super.my.package1.package2.package3.Bar.Foo

Along with removing stuff needed for reflection (code I converting doesn't use this). I've got about 2 times performance boost.

I'm using lua5.2 as target, but under LuaJIT with and without jit-compilation 2-times boost also persists.

Hope my feedback was helpful and thanks for your great work!

2016-05-29 14:16 GMT+03:00 Justin Donaldson <[hidden email]>:
Hi folks,

I recently wrote a Lua target for the Haxe language: http://haxe.org/blog/hello-lua

I think it has a lot of promise for building more sophisticated Lua apps (e.g. Nginx/lua + Redis/lua + javascript/html5/ios), while also granting the ability for those apps to integrate with other languages and frameworks.

The target is still considered to be in beta mode, but passes our extensive unit test suite.  I'd really like to get feedback from the Lua folks here on the structure/performance of the Lua generated code, and how I might better provide extern support (e.g. Haxe doesn't support multiple returns, which I know is a major sticking point at present.)

I hope some of you find it to be useful!



--
С уважением, Алимов Степан Геннадьевич.

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Haxe Lua : compile Lua/LuaJIT script from Haxe 3.3

Justin Donaldson
P.S. Can you tell me cases where reflection causes problems for you?  I can probably simplify haxe object structures if I detect Reflect is not being used.

Also, a bunch of new static analysis features were added pretty recently that removes some of the iife structures I was generating.  You should see fewer and fewer of those.

On Sun, May 29, 2016 at 3:15 PM, Justin Donaldson <[hidden email]> wrote:
The cached class concept is great idea.  I know the javascript target uses a similar method, and I'll look into making it a default for lua.  FWIW, you can easily generate "humanized" static method names with the @:export("new_method_name) metadata on the respective field.


On Sun, May 29, 2016 at 2:05 PM, stepa alimov <[hidden email]> wrote:
Hello, Justin!

> I'd really like to get feedback from the Lua folks here on the structure/performance of the Lua generated code
I've worked a bit with this generator and what I can say:
in-place increment, decrement operations converts into (function() ... end)() this is good, but not always necessary.
Some AST preprocessing could be made to detect if return of this operations are required. What I mean:
var i = 2;
i++;
print(i);

now converts into something like:

local i = 2
(function()
local hx_tmp = i
i = i + 1
return hx_tmp
end)()
print(i)

Also, long chain of table field dereferencing generates on calls, and this is major performance issue IMO.
Consider we have class Foo in package com.app.super.my.package1.package2.package3.Bar
Then, each call to static field of this class will look like

local instance = com.app.super.my.package1.package2.package3.Bar.Foo.new()

as you can see, there are 9 table lookups

For my work project I've changed generator to produce "cached" classes, so class declaration looks like:
com.app.super.my.package1.package2.package3.Bar.Foo = _hx_new();
local
com_app_super_my_package1_package2_package3_Bar_Foo =
com.app.super.my.package1.package2.package3.Bar.Foo

Along with removing stuff needed for reflection (code I converting doesn't use this). I've got about 2 times performance boost.

I'm using lua5.2 as target, but under LuaJIT with and without jit-compilation 2-times boost also persists.

Hope my feedback was helpful and thanks for your great work!

2016-05-29 14:16 GMT+03:00 Justin Donaldson <[hidden email]>:
Hi folks,

I recently wrote a Lua target for the Haxe language: http://haxe.org/blog/hello-lua

I think it has a lot of promise for building more sophisticated Lua apps (e.g. Nginx/lua + Redis/lua + javascript/html5/ios), while also granting the ability for those apps to integrate with other languages and frameworks.

The target is still considered to be in beta mode, but passes our extensive unit test suite.  I'd really like to get feedback from the Lua folks here on the structure/performance of the Lua generated code, and how I might better provide extern support (e.g. Haxe doesn't support multiple returns, which I know is a major sticking point at present.)

I hope some of you find it to be useful!



--
С уважением, Алимов Степан Геннадьевич.


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Haxe Lua : compile Lua/LuaJIT script from Haxe 3.3

stepa alimov
> Can you tell me cases where reflection causes problems for you?
Objects with proxy metatables are slightly slower.
This (https://gist.github.com/Stepets/6e079bb6163718208d5ad21191710931) simple bench shows that.
My results:
average time without metatable
init:    0.01016667    modify:    0.01614811
average time with metatable
init:    0.01118717    modify:    0.01936691

This (https://gist.github.com/Stepets/35097b0dc9ecbf90adb0843dcd5eb907) bench shows arrays lenght field update impact
My results:
average time original metatable
init:    0.00510624    modify:    0.00062112
average time simple metatable
init:    0.00358827    modify:    0.00064114
average time without metatable
init:    0.00047244    modify:    0.00064349

I'm using lua5.2


2016-05-29 16:33 GMT+03:00 Justin Donaldson <[hidden email]>:
P.S. Can you tell me cases where reflection causes problems for you?  I can probably simplify haxe object structures if I detect Reflect is not being used.

Also, a bunch of new static analysis features were added pretty recently that removes some of the iife structures I was generating.  You should see fewer and fewer of those.

On Sun, May 29, 2016 at 3:15 PM, Justin Donaldson <[hidden email]> wrote:
The cached class concept is great idea.  I know the javascript target uses a similar method, and I'll look into making it a default for lua.  FWIW, you can easily generate "humanized" static method names with the @:export("new_method_name) metadata on the respective field.


On Sun, May 29, 2016 at 2:05 PM, stepa alimov <[hidden email]> wrote:
Hello, Justin!

> I'd really like to get feedback from the Lua folks here on the structure/performance of the Lua generated code
I've worked a bit with this generator and what I can say:
in-place increment, decrement operations converts into (function() ... end)() this is good, but not always necessary.
Some AST preprocessing could be made to detect if return of this operations are required. What I mean:
var i = 2;
i++;
print(i);

now converts into something like:

local i = 2
(function()
local hx_tmp = i
i = i + 1
return hx_tmp
end)()
print(i)

Also, long chain of table field dereferencing generates on calls, and this is major performance issue IMO.
Consider we have class Foo in package com.app.super.my.package1.package2.package3.Bar
Then, each call to static field of this class will look like

local instance = com.app.super.my.package1.package2.package3.Bar.Foo.new()

as you can see, there are 9 table lookups

For my work project I've changed generator to produce "cached" classes, so class declaration looks like:
com.app.super.my.package1.package2.package3.Bar.Foo = _hx_new();
local
com_app_super_my_package1_package2_package3_Bar_Foo =
com.app.super.my.package1.package2.package3.Bar.Foo

Along with removing stuff needed for reflection (code I converting doesn't use this). I've got about 2 times performance boost.

I'm using lua5.2 as target, but under LuaJIT with and without jit-compilation 2-times boost also persists.

Hope my feedback was helpful and thanks for your great work!

2016-05-29 14:16 GMT+03:00 Justin Donaldson <[hidden email]>:
Hi folks,

I recently wrote a Lua target for the Haxe language: http://haxe.org/blog/hello-lua

I think it has a lot of promise for building more sophisticated Lua apps (e.g. Nginx/lua + Redis/lua + javascript/html5/ios), while also granting the ability for those apps to integrate with other languages and frameworks.

The target is still considered to be in beta mode, but passes our extensive unit test suite.  I'd really like to get feedback from the Lua folks here on the structure/performance of the Lua generated code, and how I might better provide extern support (e.g. Haxe doesn't support multiple returns, which I know is a major sticking point at present.)

I hope some of you find it to be useful!



--
С уважением, Алимов Степан Геннадьевич.





--
С уважением, Алимов Степан Геннадьевич.
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Haxe Lua : compile Lua/LuaJIT script from Haxe 3.3

Justin Donaldson
Quick update:

I pushed a few more changes...  I figured out a way to shave some time off the base array type using your simpler metatable method (thanks!).  It involved refactoring some other behaviors but it looks like a solid speedup of 50% or more.

I don't think I can make array any faster though... it needs to track its length in order to accommodate nil values, and I believe a __newindex metatable routine is the only way to do this.

However, there's also a haxe.ds.Vector type.  This type requires the length to be given up front in the constructor, and is common across all platforms.  I've turned it into a simple proxy table, and it preserves the speed accordingly:

    1 Main.hx|7 info| init
    2 Main.hx|13 info| 0.131187 is the value for vector
    3 Main.hx|19 info| 0.135397 is the value for table
    4 Main.hx|25 info| 1.487719 is the value for array
    5 Main.hx|27 info| ------------------------------
    6 Main.hx|28 info| modify
    7 Main.hx|35 info| 0.12161 is the value for vector
    8 Main.hx|41 info| 0.125957 is the value for table
    9 Main.hx|47 info| 1.408035 is the value for array

The other benefit is that the std vector type is fairly common on platforms that take advantage of them like (c++).  If you can deal with the constraint of a fixed length, using a vector type makes for big speedups on many targets.


On Sun, May 29, 2016 at 4:33 PM, stepa alimov <[hidden email]> wrote:
> Can you tell me cases where reflection causes problems for you?
Objects with proxy metatables are slightly slower.
This (https://gist.github.com/Stepets/6e079bb6163718208d5ad21191710931) simple bench shows that.
My results:
average time without metatable
init:    0.01016667    modify:    0.01614811
average time with metatable
init:    0.01118717    modify:    0.01936691

This (https://gist.github.com/Stepets/35097b0dc9ecbf90adb0843dcd5eb907) bench shows arrays lenght field update impact
My results:
average time original metatable
init:    0.00510624    modify:    0.00062112
average time simple metatable
init:    0.00358827    modify:    0.00064114
average time without metatable
init:    0.00047244    modify:    0.00064349

I'm using lua5.2


2016-05-29 16:33 GMT+03:00 Justin Donaldson <[hidden email]>:
P.S. Can you tell me cases where reflection causes problems for you?  I can probably simplify haxe object structures if I detect Reflect is not being used.

Also, a bunch of new static analysis features were added pretty recently that removes some of the iife structures I was generating.  You should see fewer and fewer of those.

On Sun, May 29, 2016 at 3:15 PM, Justin Donaldson <[hidden email]> wrote:
The cached class concept is great idea.  I know the javascript target uses a similar method, and I'll look into making it a default for lua.  FWIW, you can easily generate "humanized" static method names with the @:export("new_method_name) metadata on the respective field.


On Sun, May 29, 2016 at 2:05 PM, stepa alimov <[hidden email]> wrote:
Hello, Justin!

> I'd really like to get feedback from the Lua folks here on the structure/performance of the Lua generated code
I've worked a bit with this generator and what I can say:
in-place increment, decrement operations converts into (function() ... end)() this is good, but not always necessary.
Some AST preprocessing could be made to detect if return of this operations are required. What I mean:
var i = 2;
i++;
print(i);

now converts into something like:

local i = 2
(function()
local hx_tmp = i
i = i + 1
return hx_tmp
end)()
print(i)

Also, long chain of table field dereferencing generates on calls, and this is major performance issue IMO.
Consider we have class Foo in package com.app.super.my.package1.package2.package3.Bar
Then, each call to static field of this class will look like

local instance = com.app.super.my.package1.package2.package3.Bar.Foo.new()

as you can see, there are 9 table lookups

For my work project I've changed generator to produce "cached" classes, so class declaration looks like:
com.app.super.my.package1.package2.package3.Bar.Foo = _hx_new();
local
com_app_super_my_package1_package2_package3_Bar_Foo =
com.app.super.my.package1.package2.package3.Bar.Foo

Along with removing stuff needed for reflection (code I converting doesn't use this). I've got about 2 times performance boost.

I'm using lua5.2 as target, but under LuaJIT with and without jit-compilation 2-times boost also persists.

Hope my feedback was helpful and thanks for your great work!

2016-05-29 14:16 GMT+03:00 Justin Donaldson <[hidden email]>:
Hi folks,

I recently wrote a Lua target for the Haxe language: http://haxe.org/blog/hello-lua

I think it has a lot of promise for building more sophisticated Lua apps (e.g. Nginx/lua + Redis/lua + javascript/html5/ios), while also granting the ability for those apps to integrate with other languages and frameworks.

The target is still considered to be in beta mode, but passes our extensive unit test suite.  I'd really like to get feedback from the Lua folks here on the structure/performance of the Lua generated code, and how I might better provide extern support (e.g. Haxe doesn't support multiple returns, which I know is a major sticking point at present.)

I hope some of you find it to be useful!



--
С уважением, Алимов Степан Геннадьевич.





--
С уважением, Алимов Степан Геннадьевич.

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Haxe Lua : compile Lua/LuaJIT script from Haxe 3.3

Javier Guerra Giraldez
On 31 May 2016 at 05:42, Justin Donaldson <[hidden email]> wrote:
> it needs to track its length in order to accommodate nil values


The usual approach to store nil values in a table is to use your own
'pseudo nil' value.

------
local mynil = {}

function set(t,k,v)
    if v == nil then v = mynil end
    t[k] = v
end

function get(t,k)
    local v = t[k]
    if v == mynil then v = nil end
    return v
end
-------



--
Javier

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Haxe Lua : compile Lua/LuaJIT script from Haxe 3.3

Andrei Radu
Hi, is there any way to generate line mappings? It would be really useful for debugging purposes. 

Thx,
Andrei

On Tue, May 31, 2016 at 11:19 AM, Javier Guerra Giraldez <[hidden email]> wrote:
On 31 May 2016 at 05:42, Justin Donaldson <[hidden email]> wrote:
> it needs to track its length in order to accommodate nil values


The usual approach to store nil values in a table is to use your own
'pseudo nil' value.

------
local mynil = {}

function set(t,k,v)
    if v == nil then v = mynil end
    t[k] = v
end

function get(t,k)
    local v = t[k]
    if v == mynil then v = nil end
    return v
end
-------



--
Javier


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Haxe Lua : compile Lua/LuaJIT script from Haxe 3.3

Justin Donaldson
In reply to this post by Javier Guerra Giraldez


On Tue, May 31, 2016 at 10:19 AM, Javier Guerra Giraldez <[hidden email]> wrote:
On 31 May 2016 at 05:42, Justin Donaldson <[hidden email]> wrote:
> it needs to track its length in order to accommodate nil values


The usual approach to store nil values in a table is to use your own
'pseudo nil' value.

Consider creating an array [1,2,3], and then setting the index 100 to 4.  The indexes 4-99 will still be nil.   The length of that array reported by maxn/# would still be indeterminate.  How do I resolve that?



Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Haxe Lua : compile Lua/LuaJIT script from Haxe 3.3

Justin Donaldson
In reply to this post by Andrei Radu
On Tue, May 31, 2016 at 12:52 PM, Andrei Radu <[hidden email]> wrote:
Hi, is there any way to generate line mappings? It would be really useful for debugging purposes. 


What kind of line mappings do you need?  Is there a standard format?

FWIW any trace() statement will generate line numbers according to the original haxe source.  I use that for debugging.


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Haxe Lua : compile Lua/LuaJIT script from Haxe 3.3

Soni "They/Them" L.
In reply to this post by Justin Donaldson


On 31/05/16 02:00 PM, Justin Donaldson wrote:

>
>
> On Tue, May 31, 2016 at 10:19 AM, Javier Guerra Giraldez
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     On 31 May 2016 at 05:42, Justin Donaldson <[hidden email]
>     <mailto:[hidden email]>> wrote:
>     > it needs to track its length in order to accommodate nil values
>
>
>     The usual approach to store nil values in a table is to use your own
>     'pseudo nil' value.
>
>
> Consider creating an array [1,2,3], and then setting the index 100 to
> 4.  The indexes 4-99 will still be nil. The length of that array
> reported by maxn/# would still be indeterminate.  How do I resolve that?
>
>
>
With a little help from C, it's possible to create a very efficient
coroutine-based nil-safe array. Just need a function exchange(index,
value, ...) -> ... (that's a varargs and a varret, respectively), which
changes the value in `...` at index `index` with `value` and returns the
result. Everything else can be done from Lua. It works better as a
stack, however.

Length operation is select('#', ...), push is `val, ...` and pop is
`select(2, ...)`. See http://sprunge.us/CGCT

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Haxe Lua : compile Lua/LuaJIT script from Haxe 3.3

Andrei Radu
In reply to this post by Justin Donaldson


What kind of line mappings do you need?  Is there a standard format?

FWIW any trace() statement will generate line numbers according to the original haxe source.  I use that for debugging.



I need line mappings for running a lua debugger on Haxe code. It's similar to what CoffeScript/TypeScript, etc do to allow JS debugging, and you can find an example format here: http://coffeescript.org/documentation/docs/sourcemap.html
It shouldn't be difficult for me to hack it, as long as the AST keeps track of the line/column in the source that generated each node. 


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Haxe Lua : compile Lua/LuaJIT script from Haxe 3.3

Justin Donaldson
As an update, the video from my talk on Haxe and Lua is available : 

On Tue, May 31, 2016 at 12:30 PM, Andrei Radu <[hidden email]> wrote:


What kind of line mappings do you need?  Is there a standard format?

FWIW any trace() statement will generate line numbers according to the original haxe source.  I use that for debugging.



I need line mappings for running a lua debugger on Haxe code. It's similar to what CoffeScript/TypeScript, etc do to allow JS debugging, and you can find an example format here: http://coffeescript.org/documentation/docs/sourcemap.html
It shouldn't be difficult for me to hack it, as long as the AST keeps track of the line/column in the source that generated each node.