Loading DSL Data

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Loading DSL Data

Marc Lepage
Hi, looking for some thoughts on loading DSL style data.

I have it working but would like to swap out the environment when doing so, so it doesn't trample on the global environment.

I have something like:

function load(name)
    local env = getfenv(0)
    setfenv(0, dsl)
    local data = require(name)
    setfenv(0, env)
end

Where I get the thread's environment, set the dsl table (with all the DSL functions) as the environment, load the data (using require), then restore the environment (presumably the _G table). Of course I should really pcall the load instead of require, and return the data, but that's the idea.

This is using Lua 5.1.5. I understand that Lua 5.2 changes the way environments are handled?

If anyone could offer tips here or even better, point to a page which explains a lot of this in this context, that would be super great.

Thanks,
Marc
Reply | Threaded
Open this post in threaded view
|

Re: Loading DSL Data

Hisham
On 20 May 2014 01:31, Marc Lepage <[hidden email]> wrote:

> Hi, looking for some thoughts on loading DSL style data.
>
> I have it working but would like to swap out the environment when doing so,
> so it doesn't trample on the global environment.
>
> I have something like:
>
> function load(name)
>     local env = getfenv(0)
>     setfenv(0, dsl)
>     local data = require(name)
>     setfenv(0, env)
> end
>
> Where I get the thread's environment, set the dsl table (with all the DSL
> functions) as the environment, load the data (using require), then restore
> the environment (presumably the _G table). Of course I should really pcall
> the load instead of require, and return the data, but that's the idea.
>
> This is using Lua 5.1.5. I understand that Lua 5.2 changes the way
> environments are handled?
>
> If anyone could offer tips here or even better, point to a page which
> explains a lot of this in this context, that would be super great.

it might or might not be exactly what you're looking for but your
problem reminded me of this:

https://github.com/keplerproject/luarocks/blob/master/src/luarocks/persist.lua#L19-L58

Hope that helps!

-- Hisham

Reply | Threaded
Open this post in threaded view
|

Re: Loading DSL Data

Marc Lepage
On Tue, May 20, 2014 at 12:40 AM, Hisham <[hidden email]> wrote:
On 20 May 2014 01:31, Marc Lepage <[hidden email]> wrote:
> Hi, looking for some thoughts on loading DSL style data.
>
> I have it working but would like to swap out the environment when doing so,
> so it doesn't trample on the global environment.
>
> I have something like:
>
> function load(name)
>     local env = getfenv(0)
>     setfenv(0, dsl)
>     local data = require(name)
>     setfenv(0, env)
> end
>
> Where I get the thread's environment, set the dsl table (with all the DSL
> functions) as the environment, load the data (using require), then restore
> the environment (presumably the _G table). Of course I should really pcall
> the load instead of require, and return the data, but that's the idea.
>
> This is using Lua 5.1.5. I understand that Lua 5.2 changes the way
> environments are handled?
>
> If anyone could offer tips here or even better, point to a page which
> explains a lot of this in this context, that would be super great.

it might or might not be exactly what you're looking for but your
problem reminded me of this:

https://github.com/keplerproject/luarocks/blob/master/src/luarocks/persist.lua#L19-L58

Hope that helps!

-- Hisham


Thanks. My first kick at the can just set a metatable on _G during the load, to try to avoid setting some functions. All the DSL functions were global.

That was messy (as I knew) so the last couple hours I managed to clean it up as follows.

All DSL functions exist in a table dsl. When loading, an empty environment env is used (via setfenv(0)). The empty environment has a metatable which directs __index access to the DSL table. That is how DSL functions are found. In fact, only DSL functions can be found (since the environment is otherwise empty), I'm OK with that.

The DSL table has a metatable which directs __index to a proxy table. That also starts out empty.

The environment metatable has __newindex which filters out certain attempts to set globals, and does something special with them. The rest are stored into the proxy table.

This means I cannot overwrite my DSL functions. E.g. if dsl.foo is a function, and I set global foo, it goes into the proxy table, and any access of foo still returns dsl.foo.

The point of this is that later on, after loading the DSL, the environment will switch back and I will also dump the contents of proxy back into it. So, when done using the DSL, dsl.foo no longer exists, and that foo that was set as a global is now a global.

That's the idea, anyways. I've hit some sort of hitch with accesses to globals in my "global" foo function no longer working after all this. Maybe I need to learn more about function environments.

Anyways this is almost working as I desire, just some glitches to work out. Still interested in tips.

Marc
Reply | Threaded
Open this post in threaded view
|

Re: Loading DSL Data

Marc Lepage
On Tue, May 20, 2014 at 1:59 AM, Marc Lepage <[hidden email]> wrote:
On Tue, May 20, 2014 at 12:40 AM, Hisham <[hidden email]> wrote:
On 20 May 2014 01:31, Marc Lepage <[hidden email]> wrote:
> Hi, looking for some thoughts on loading DSL style data.
>
> I have it working but would like to swap out the environment when doing so,
> so it doesn't trample on the global environment.
>
> I have something like:
>
> function load(name)
>     local env = getfenv(0)
>     setfenv(0, dsl)
>     local data = require(name)
>     setfenv(0, env)
> end
>
> Where I get the thread's environment, set the dsl table (with all the DSL
> functions) as the environment, load the data (using require), then restore
> the environment (presumably the _G table). Of course I should really pcall
> the load instead of require, and return the data, but that's the idea.
>
> This is using Lua 5.1.5. I understand that Lua 5.2 changes the way
> environments are handled?
>
> If anyone could offer tips here or even better, point to a page which
> explains a lot of this in this context, that would be super great.

it might or might not be exactly what you're looking for but your
problem reminded me of this:

https://github.com/keplerproject/luarocks/blob/master/src/luarocks/persist.lua#L19-L58

Hope that helps!

-- Hisham


Thanks. My first kick at the can just set a metatable on _G during the load, to try to avoid setting some functions. All the DSL functions were global.

That was messy (as I knew) so the last couple hours I managed to clean it up as follows.

All DSL functions exist in a table dsl. When loading, an empty environment env is used (via setfenv(0)). The empty environment has a metatable which directs __index access to the DSL table. That is how DSL functions are found. In fact, only DSL functions can be found (since the environment is otherwise empty), I'm OK with that.

The DSL table has a metatable which directs __index to a proxy table. That also starts out empty.

The environment metatable has __newindex which filters out certain attempts to set globals, and does something special with them. The rest are stored into the proxy table.

This means I cannot overwrite my DSL functions. E.g. if dsl.foo is a function, and I set global foo, it goes into the proxy table, and any access of foo still returns dsl.foo.

The point of this is that later on, after loading the DSL, the environment will switch back and I will also dump the contents of proxy back into it. So, when done using the DSL, dsl.foo no longer exists, and that foo that was set as a global is now a global.

That's the idea, anyways. I've hit some sort of hitch with accesses to globals in my "global" foo function no longer working after all this. Maybe I need to learn more about function environments.

Anyways this is almost working as I desire, just some glitches to work out. Still interested in tips.

Marc


A wee bit of investigation has revealed that the issue is that even after swapping back the environment at the 0 (global) level with setfenv, the environment of the data that I loaded is still the DSL-is-loading environment with all the special stuff, and no access to globals. That is a problem. Like I said, I need to learn more about how to properly swap these environments out (and then I suppose learn it again for Lua 5.2).

Marc
Reply | Threaded
Open this post in threaded view
|

Re: Loading DSL Data

Choonster TheMage
In reply to this post by Marc Lepage
On 20 May 2014 14:31, Marc Lepage <[hidden email]> wrote:
Hi, looking for some thoughts on loading DSL style data.

I have it working but would like to swap out the environment when doing so, so it doesn't trample on the global environment.

I have something like:

function load(name)
    local env = getfenv(0)
    setfenv(0, dsl)
    local data = require(name)
    setfenv(0, env)
end

Where I get the thread's environment, set the dsl table (with all the DSL functions) as the environment, load the data (using require), then restore the environment (presumably the _G table). Of course I should really pcall the load instead of require, and return the data, but that's the idea.

This is using Lua 5.1.5. I understand that Lua 5.2 changes the way environments are handled?

If anyone could offer tips here or even better, point to a page which explains a lot of this in this context, that would be super great.

Thanks,
Marc

Could you load the file with `loadfile` instead of `require` and then set the environment of the loaded chunk with `setfenv` before executing it? If you need to turn a module name into a full path, you can use an implementation of Lua 5.2's `package.searchpath` function like the one from Steve Donovan's Penlight compat module[1].

If/when you move to Lua 5.2, you can pass the environment table directly to `loadfile` instead of using `setfenv`.

Regards,
Choonster

Reply | Threaded
Open this post in threaded view
|

Re: Loading DSL Data

Thierry@spoludo
In reply to this post by Marc Lepage
‎Hi,

Just a question: why the DSL has side effects on the upper scope?

‎I understand that the idea is to use the keywords without scoping. I tend not to do this anymore rather using scope: that allows having a local context. This is not so unusual with XML schemas, xs:...

If I don't need these contexts, I still scope the language ‎and use local var to unscope rather than using env.

local mydsl= require'my.dsl.grammar'
‎local kw1, kw2 = mydsl.kw1, mydsl.kw2 -- easy copy/paste

How your grammar ‎looks like?

Https://bitbucket.org/spoludo/tools/src

From: Marc Lepage
Sent: mardi 20 mai 2014 07:32
To: Lua mailing list
Reply To: Lua mailing list
Subject: Loading DSL Data

Hi, looking for some thoughts on loading DSL style data.

I have it working but would like to swap out the environment when doing so, so it doesn't trample on the global environment.


Reply | Threaded
Open this post in threaded view
|

Re: Loading DSL Data

Marc Lepage
On Tue, May 20, 2014 at 7:47 AM, Thierry@spoludo <[hidden email]> wrote:
‎Hi,

Just a question: why the DSL has side effects on the upper scope?

‎I understand that the idea is to use the keywords without scoping. I tend not to do this anymore rather using scope: that allows having a local context. This is not so unusual with XML schemas, xs:...

If I don't need these contexts, I still scope the language ‎and use local var to unscope rather than using env.

local mydsl= require'my.dsl.grammar'
‎local kw1, kw2 = mydsl.kw1, mydsl.kw2 -- easy copy/paste

How your grammar ‎looks like?


From: Marc Lepage
Sent: mardi 20 mai 2014 07:32
To: Lua mailing list
Reply To: Lua mailing list
Subject: Loading DSL Data

Hi, looking for some thoughts on loading DSL style data.

I have it working but would like to swap out the environment when doing so, so it doesn't trample on the global environment.



Hi Thierry, that is a good question.

Firstly, I've been experimenting lately with injecting stuff into the global table from loaded data, in both this and another project. So far it's looking promising, it simplifies a lot. Basically, things have IDs, and for convenience I make a global using that ID. Since these are small projects, it's no big deal.

In this case, though, some of the "keywords" have different meaning (or at least must do the same thing) depending on their form. Here's an example:

object 'A'
    foo 'bar'
object 'B'
    function foo() ... end

This allows me to have a simple foo on one object, but to make it a routine in another object, for more complex behaviour. In practical terms, these are just four function calls (regardless of indentation). I don't have any nesting so this is fine.

Function foo (in the DSL) handles the simple string case. So I can't have the second object's definition of function foo overwrite it as a new global. That's the thing I trap, instead to set the function as a property of the object being built.

Where it gets complicated is, some of my DSL keywords that build objects while loading data, are also global functions in the engine outside the DSL and its data. But they need to be accessible when the functions in the DSL-loaded objects are run. Something like this:

object 'C'
    function foo() -- this is object C's foo
        foo() -- it should be able to call global foo
   end

I know it doesn't make a lot of sense when it's named "foo" but trust me with real names it becomes quite sensible. :-)

I also know I could scope the DSL's keywords but I am trying to avoid that. I just want a really clean easy syntax, in part because I would like children to be able to produce the data files. The less distraction the better.

I have it all working except for some environment swapping issues when loading the data, which I've now figured out, so it all should be fine. And again, part of this is experimentation, I may later decide that I don't like all the globals. I will let everything play out and see what conclusion I come to after I have built more of this project.

Thanks for asking,
Marc
Reply | Threaded
Open this post in threaded view
|

Re: Loading DSL Data

Marc Lepage
In reply to this post by Choonster TheMage
On Tue, May 20, 2014 at 6:05 AM, Choonster TheMage <[hidden email]> wrote:
On 20 May 2014 14:31, Marc Lepage <[hidden email]> wrote:
Hi, looking for some thoughts on loading DSL style data.

I have it working but would like to swap out the environment when doing so, so it doesn't trample on the global environment.

I have something like:

function load(name)
    local env = getfenv(0)
    setfenv(0, dsl)
    local data = require(name)
    setfenv(0, env)
end

Where I get the thread's environment, set the dsl table (with all the DSL functions) as the environment, load the data (using require), then restore the environment (presumably the _G table). Of course I should really pcall the load instead of require, and return the data, but that's the idea.

This is using Lua 5.1.5. I understand that Lua 5.2 changes the way environments are handled?

If anyone could offer tips here or even better, point to a page which explains a lot of this in this context, that would be super great.

Thanks,
Marc

Could you load the file with `loadfile` instead of `require` and then set the environment of the loaded chunk with `setfenv` before executing it? If you need to turn a module name into a full path, you can use an implementation of Lua 5.2's `package.searchpath` function like the one from Steve Donovan's Penlight compat module[1].

If/when you move to Lua 5.2, you can pass the environment table directly to `loadfile` instead of using `setfenv`.

Regards,
Choonster



Thanks, Choonster.

Further reading late last night brought me to this answer, which has essentially the same suggestion (to loadfile, setfenv on the chunk, then call it).

I also found some code here
which checks the Lua version and in the case of 5.2, uses the newer (easier) technique you described.

Finally, a comment here
reveals the reason why my initial attempt didn't work, is that the functions use the environment they were compiled with, not the environment they were called from.

So I think it's all clear now and I've learned a reasonable amount about swapping function environments. I had the idea mostly correct but got tripped up in some details.

Cheers.
Marc