Lua as a configuration language -- validation

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Lua as a configuration language -- validation

Chris Smith
I’m using Lua for configuration.  That means I need to validate entries for correctness, provide default values and warning messages where necessary, as well as ensuring my code doesn’t blow up when trying to access a non-existent structure.  I’m finding my code becoming bloated with largely boiler-plate safety code, particularly when accessing entries that are nested a few levels deep.

Is there a nice, idiomatic way of handling this?

Regards,
Chris

Chris Smith <[hidden email]>



Reply | Threaded
Open this post in threaded view
|

Re: Lua as a configuration language -- validation

Luiz Henrique de Figueiredo
Please show us a sample configuration file and what validation you have in mind.

Reply | Threaded
Open this post in threaded view
|

Re: Lua as a configuration language -- validation

Chris Smith

> On 22 Aug 2019, at 13:58, Luiz Henrique de Figueiredo <[hidden email]> wrote:
>
> Please show us a sample configuration file and what validation you have in mind.


Ok, well let’s take a trivial and contrived example of a UI configuration.  The configuration might look something like this:

UI = {
  mainwindow = {
    size = {
      width = 100,
      height = 200
    }
  }
}

Now, my code wants to use this information very simply:

local width = config.UI.mainwindow.size.width

But before I can do that, as a minimum I need to verify the structure and format of the config before I can safely use it.  Very naively, I might do something like this:

local width
if type(UI) == ’table’ then
  if type(UI.mainwindow) == ’table’ then
    if type(UI.mainwindow.size) == ’table’ then
      — now can access safely
      width = UI.mainwindow.size.width
    end
  end
end

If type(width) == ’number’ then
  — do something
end

That’s an awful lot of cruft just for one element, and I haven’t even dealt with range checking, default values and passing back warnings to the user.  There are a few ways I can think to make it simpler, like wrapping a simple access in a pcall to see if it crashes or not, but this must be quite a common issue and I wondered if there are already some standard libraries or patterns to deal with it before I start re-inventing wheels.

Regards,
Chris

Chris Smith <[hidden email]>

Reply | Threaded
Open this post in threaded view
|

Re: Lua as a configuration language -- validation

Coda Highland
In reply to this post by Chris Smith


On Thu, Aug 22, 2019 at 7:26 AM Chris Smith <[hidden email]> wrote:
I’m using Lua for configuration.  That means I need to validate entries for correctness, provide default values and warning messages where necessary, as well as ensuring my code doesn’t blow up when trying to access a non-existent structure.  I’m finding my code becoming bloated with largely boiler-plate safety code, particularly when accessing entries that are nested a few levels deep.

Is there a nice, idiomatic way of handling this?

Regards,
Chris

Chris Smith <[hidden email]>


Default values in top-level structures are fairly straightforward: prepopulate a table with the defaults and pass it as the third ("env") parameter to loadfile(). This has the advantage that the user's configuration file can read the defaults and perform logic on them if necessary. (But note the caveat below.)

Default values in nested structures can be handled with a function akin to Javascript's Object.assign() -- construct a deep copy of a table containing the default values, and use a single generic function that recursively copies keys from the object coming from the configuration file into the default-populated table. (By "recursively", I mean that if the key exists in both tables, and the value is a nested table in both tables, then call the same function using those values.)

One possible technique for validation to consider would be to construct a table with the same layout as the configuration. The values in this table would be validation functions. You can then use a generic, similarly recursive function to iterate over the validation table and the configuration table and run the validator functions on the corresponding values.

Now for the caveat I mentioned: Using plain Lua scripts for configuration can be a security risk if your threat model includes malicious configuration files. If your threat model assumes that authorized administrators are the only ones that can modify the files, then you don't necessarily need to worry about this (although it might be a good idea to put some safeties in place to protect against accidents) but if it's a concern then you'll need to set up a sandbox.

Given the above techniques, your example from your other email might correspond to something like this:

defaults = {
  UI = { mainwindow = { size = { width = 640, height = 480 } } }
}

function validateSize(value)
  return type(value) == 'table' and type(value.width) == 'number' and type(value.height) == 'number'
end

validators = {
  UI = {
    mainwindow = {
      size = validateSize
    }
  }
}

/s/ Adam
Reply | Threaded
Open this post in threaded view
|

Re: Lua as a configuration language -- validation

Scott Morgan
In reply to this post by Chris Smith
On 22/08/2019 13:25, Chris Smith wrote:
> I’m using Lua for configuration.  That means I need to validate
> entries for correctness, provide default values and warning messages
> where necessary, as well as ensuring my code doesn’t blow up when
> trying to access a non-existent structure.  I’m finding my code
> becoming bloated with largely boiler-plate safety code, particularly
> when accessing entries that are nested a few levels deep.
>
> Is there a nice, idiomatic way of handling this?

Would this help?

http://lua-users.org/lists/lua-l/2015-07/msg00138.html

Whole thread might be of interest:
http://lua-users.org/lists/lua-l/2015-07/msg00119.html

Scott

Reply | Threaded
Open this post in threaded view
|

Re: Lua as a configuration language -- validation

Frank Kastenholz-2
In reply to this post by Chris Smith
Hi

A long time ago I wrote a simple function to validate the contents of a lua table.

It took as input the table to validate and a table describing the table. The descriptor table had an entry for each possible entry in the data table.  For each possible entry it indicated a) whether the entry was required or optional, the valid type()s of the entry, and a bunch of stuff to validate the data (eg, mom and max numerical values).  If the entry  can itself be a table then the descriptor table has a reference to another descriptor table (etc)

The checking function also took a flag to indicate whether the table being validated could contain entries not in the descriptor table (or must only have entries in the descriptor table).


I don’t have access to it right this moment ... I can dig it out tonight and clean it up and post it tonight...

Frank



> On Aug 22, 2019, at 8:25 AM, Chris Smith <[hidden email]> wrote:
>
> I’m using Lua for configuration.  That means I need to validate entries for correctness, provide default values and warning messages where necessary, as well as ensuring my code doesn’t blow up when trying to access a non-existent structure.  I’m finding my code becoming bloated with largely boiler-plate safety code, particularly when accessing entries that are nested a few levels deep.
>
> Is there a nice, idiomatic way of handling this?
>
> Regards,
> Chris
> —
> Chris Smith <[hidden email]>
>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Lua as a configuration language -- validation

szbnwer@gmail.com
In reply to this post by Scott Morgan
hi there! :)

u can use something like `read(root, path)` (as well as similarly
`write()` and `delete()`) that will give back either the wished result
or info about the last existing element; or a fallback mechanism that
will try to look up the same path from more than one root elements in
case of need; and its best if u make some freestanding value checker
functions. then combine these, and they are universal enough to use
them kinda much anywhere else...

all the bests! :)

Reply | Threaded
Open this post in threaded view
|

Re: Lua as a configuration language -- validation

Luiz Henrique de Figueiredo
In reply to this post by Chris Smith
Let Lua do the work. See
http://lua-users.org/lists/lua-l/2006-12/msg00575.html and others in
that thread for instance.

Reply | Threaded
Open this post in threaded view
|

Re: Lua as a configuration language -- validation

Russell Haley


On Thu, Aug 22, 2019 at 8:32 AM Luiz Henrique de Figueiredo <[hidden email]> wrote:
Let Lua do the work. See
http://lua-users.org/lists/lua-l/2006-12/msg00575.html and others in
that thread for instance.


From a brief look at Mr. Ramseys' answer, creating a similar solution might be possible with the Moses library. The functions containsKeys, sameKeys and map all look like they offer some compelling tools...

 
Reply | Threaded
Open this post in threaded view
|

Re: Lua as a configuration language -- validation

Russell Haley


On Thu, Aug 22, 2019 at 1:17 PM Russell Haley <[hidden email]> wrote:


On Thu, Aug 22, 2019 at 8:32 AM Luiz Henrique de Figueiredo <[hidden email]> wrote:
Let Lua do the work. See
http://lua-users.org/lists/lua-l/2006-12/msg00575.html and others in
that thread for instance.


From a brief look at Mr. Ramseys' answer, creating a similar solution might be possible with the Moses library. The functions containsKeys, sameKeys and map all look like they offer some compelling tools...

 
Sorry, forgot to include links to Moses:


 
Reply | Threaded
Open this post in threaded view
|

Re: Lua as a configuration language -- validation

aryajur
In reply to this post by Chris Smith


Ok, well let’s take a trivial and contrived example of a UI configuration.  The configuration might look something like this:

UI = {
  mainwindow = {
    size = {
      width = 100,
      height = 200
    }
  }
}

Now, my code wants to use this information very simply:

local width = config.UI.mainwindow.size.width

But before I can do that, as a minimum I need to verify the structure and format of the config before I can safely use it.  Very naively, I might do something like this:

local width
if type(UI) == ’table’ then
  if type(UI.mainwindow) == ’table’ then
    if type(UI.mainwindow.size) == ’table’ then
      — now can access safely
      width = UI.mainwindow.size.width
    end
  end
end


 I can't find the thread now but I think Luiz suggested a technique like:

debug.setmetatable(nil,{__index = function(t,k) return nil end})
now you can just do
t = {}
t.x   -- nil
t.x.a -- nil
t.x.a.b -- nil

So now you can just load the config and do:
width = UI.mainwindow.size.width or 23 (or whatever is the default) 


Reply | Threaded
Open this post in threaded view
|

Re: Lua as a configuration language -- validation

kurapica
In reply to this post by Chris Smith

 

In my lib - https://github.com/kurapica/PLoop, there is a structure system can be used to do those type validation, nest, default, require and others, for your example, its like

 

require "PLoop" (function(_ENV)

            struct "UI" {

                        mainwindow = {

                                    require = true,

                                    type = struct {

                                                size = {

                                                            require = true,

                                                            type = struct {

                                                                        width = { require = true, Number },

                                                                        height = { require = true, Number },

                                                            }

                                                }

                                    }

                        }

            }

 

            -- Error: xxx.lua:18: Usage: UI(mainwindow) - the mainwindow.size.height can't be nil

            ui = UI { mainwindow = { size = { width = 200 } } }

end)

 

You can also define template struct like Range

 

require "PLoop" (function(_ENV)

            __Arguments__{ Number, Number }

            struct "Range" (function(_ENV, min, max)

                        __base = Number

 

                        function __valid(val)

                                    if val < min or val >  max then

                                                return "The %s must bwtween " .. min .. " and " .. max

                                    end

                        end

            end)

 

            struct "Size" {

                        -- it's a little weird, but I have no other way to pass multi template params

                        width = Range[{0, 400}]

            }

 

            -- Usage: Size(width) - The width must bwtween 0 and 400

            v = Size{ width = 700 }

end)

 

 

Reply | Threaded
Open this post in threaded view
|

Re: Lua as a configuration language -- validation

Chris Smith
Thank you for all the responses.  I think for my current purpose I’m going to go with a simple pcall wrapper.  Something like this:

local value, error = config(function() return config.UI.mainwindow.size.width end, ’number’, 100)

Or this:

local value, error = config(‘config.UI.mainwindow.size.width’, ’number’, 100)

The second would need a load() before pcall(), but makes it much easier to construct an error message.

Regards,
Chris

Chris Smith <[hidden email]>



Reply | Threaded
Open this post in threaded view
|

Re: Lua as a configuration language -- validation

szbnwer@gmail.com
hi again! :)

just a footnote to my previous message:
by `path`, ive meant `{'a', 'b', 'c'}` so u dont need pcall but a
lookup function will reach the wished point. it can be made
bulletproof, and there is no need to magic, like `load()` and
`pcall()`, while u can also have pretty error messages and the point
where u ran out of ur path... but do whatever u want! :D

bests! :)

Reply | Threaded
Open this post in threaded view
|

Re: Lua as a configuration language -- validation

Diab Jerius-2
In reply to this post by Chris Smith
On Thu, Aug 22, 2019 at 8:25 AM Chris Smith <[hidden email]> wrote:
>
> I’m using Lua for configuration.  That means I need to validate entries for correctness, provide default values and warning messages where necessary, as well as ensuring my code doesn’t blow up when trying to access a non-existent structure.  I’m finding my code becoming bloated with largely boiler-plate safety code, particularly when accessing entries that are nested a few levels deep.
>
> Is there a nice, idiomatic way of handling this?
>


I'm not sure how idiomatic this is, but here's something I wrote up a
few years ago:

https://luarocks.org/modules/luarocks/validate-args