Proposal: package.(c)path as tables

classic Classic list List threaded Threaded
28 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Proposal: package.(c)path as tables

Daurnimator
At the moment, package.path and package.cpath are strings containing
semicolon separated lists.
I'd like to propose that they are transitioned to be sequences to
allow for easier programmatic manipulation.

The environment variables LUA_PATH, LUA_PATH_5_4, LUA_CPATH, etc
wouldn't change: they'd just need to be parsed when the package
library is loaded.

In the next release we can allow package.path and package.cpath to be
strings for compatibility.
However the construct `package.path = package.path ..
";/my/path/?.lua" would break and need to be changed to
`table.insert(package.path, "/my/path/?.lua")`
If this breakage is too much, perhaps we could have three phases:
  - phase 1. we allow table form, but still use string form on lua
command line application load.
  - phase 2. we change the default to be table form; string form is deprecated
  - phase 3. we remove support for string form.

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: package.(c)path as tables

Dirk Laurie-2
2017-09-06 10:26 GMT+02:00 Daurnimator <[hidden email]>:

> At the moment, package.path and package.cpath are strings containing
> semicolon separated lists.
> I'd like to propose that they are transitioned to be sequences to
> allow for easier programmatic manipulation.
>
> The environment variables LUA_PATH, LUA_PATH_5_4, LUA_CPATH, etc
> wouldn't change: they'd just need to be parsed when the package
> library is loaded.
>
> In the next release we can allow package.path and package.cpath to be
> strings for compatibility.
> However the construct `package.path = package.path ..
> ";/my/path/?.lua" would break and need to be changed to
> `table.insert(package.path, "/my/path/?.lua")`
> If this breakage is too much, perhaps we could have three phases:
>   - phase 1. we allow table form, but still use string form on lua
> command line application load.
>   - phase 2. we change the default to be table form; string form is deprecated
>   - phase 3. we remove support for string form.

local path = {}
package.path:gsub("[^;]*,function(x) path[#path+1]=x end

A breaking change to save having to code that? DMML

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: package.(c)path as tables

Martin
On 09/06/2017 12:32 PM, Dirk Laurie wrote:

> 2017-09-06 10:26 GMT+02:00 Daurnimator <[hidden email]>:
>> At the moment, package.path and package.cpath are strings containing
>> semicolon separated lists.
>> I'd like to propose that they are transitioned to be sequences to
>> allow for easier programmatic manipulation.
>
> local path = {}
> package.path:gsub("[^;]*,function(x) path[#path+1]=x end
>
> A breaking change to save having to code that? DMML
>

String format for <package.path> looks like fossil from early versions.
To get terms (for example, when searching module "by hands") we have to
implement our own parsing functions.

So I am completely agree with proposal.

If this change breaks existing code - new config tables may be named
like "package.path_t" and "package.cpath_t".

--[[
Also there is weird stuff in <package.config>. (String which must
contain five lines with special meaning.) I'd like to change it too.
]]

-- Martin

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: package.(c)path as tables

Charles Heywood
`package.path_t` etc. is actually quite similar to something I've done with FusionScript. I made it so FusionScript's package loader has a `package.fusepath_t` - however, I didn't make a `package.fusepath` because that would mean similarities with the other `package.c?path` variables. :+1: for `package.c?path_t`, but I'm not sure how that would work while still having a `package.path` - perhaps a special __index function that "renders" it based on the `_t` version?.

On Wed, Sep 6, 2017 at 1:09 PM Martin <[hidden email]> wrote:
On 09/06/2017 12:32 PM, Dirk Laurie wrote:
> 2017-09-06 10:26 GMT+02:00 Daurnimator <[hidden email]>:
>> At the moment, package.path and package.cpath are strings containing
>> semicolon separated lists.
>> I'd like to propose that they are transitioned to be sequences to
>> allow for easier programmatic manipulation.
>
> local path = {}
> package.path:gsub("[^;]*,function(x) path[#path+1]=x end
>
> A breaking change to save having to code that? DMML
>

String format for <package.path> looks like fossil from early versions.
To get terms (for example, when searching module "by hands") we have to
implement our own parsing functions.

So I am completely agree with proposal.

If this change breaks existing code - new config tables may be named
like "package.path_t" and "package.cpath_t".

--[[
Also there is weird stuff in <package.config>. (String which must
contain five lines with special meaning.) I'd like to change it too.
]]

-- Martin

--
--
Ryan <[hidden email]>
Software Developer / System Administrator
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: package.(c)path as tables

nobody
In reply to this post by Martin
On 2017-09-06 20:03, Martin wrote:
> To get terms (for example, when searching module "by hands") we have
> to implement our own parsing functions.

What did you need to do that `package.searchpath`[1] wasn't enough for?

-- nobody


[1]: https://www.lua.org/manual/5.3/manual.html#pdf-package.searchpath

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: package.(c)path as tables

Soni "They/Them" L.
In reply to this post by Daurnimator


On 2017-09-06 05:26 AM, Daurnimator wrote:

> At the moment, package.path and package.cpath are strings containing
> semicolon separated lists.
> I'd like to propose that they are transitioned to be sequences to
> allow for easier programmatic manipulation.
>
> The environment variables LUA_PATH, LUA_PATH_5_4, LUA_CPATH, etc
> wouldn't change: they'd just need to be parsed when the package
> library is loaded.
>
> In the next release we can allow package.path and package.cpath to be
> strings for compatibility.
> However the construct `package.path = package.path ..
> ";/my/path/?.lua" would break and need to be changed to
> `table.insert(package.path, "/my/path/?.lua")`
> If this breakage is too much, perhaps we could have three phases:
>    - phase 1. we allow table form, but still use string form on lua
> command line application load.
>    - phase 2. we change the default to be table form; string form is deprecated
>    - phase 3. we remove support for string form.
>

I'd rather them get escape sequences so I can use weird filenames with
them. (see also: not being able to use ':' in PATH)

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.


Reply | Threaded
Open this post in threaded view
|

Re: Proposal: package.(c)path as tables

Dirk Laurie-2
In reply to this post by Martin
2017-09-06 20:03 GMT+02:00 Martin <[hidden email]>:

> On 09/06/2017 12:32 PM, Dirk Laurie wrote:
>> 2017-09-06 10:26 GMT+02:00 Daurnimator <[hidden email]>:
>>> At the moment, package.path and package.cpath are strings containing
>>> semicolon separated lists.
>>> I'd like to propose that they are transitioned to be sequences to
>>> allow for easier programmatic manipulation.
>>
>> local path = {}
>> package.path:gsub("[^;]*,function(x) path[#path+1]=x end
>>
>> A breaking change to save having to code that? DMML
>>
>
> String format for <package.path> looks like fossil from early versions.

You should not lightly make remarks like that. Lua does deliver "arg"
as a table, so the designers do take that possibility into account.

The reason for the string is that operating systems do it that way.
One gsub, and your package.path turns into the $PATH that the
operating system requires. It is the obvious and easy way to provide
the interface.

> To get terms (for example, when searching module "by hands") we have to
> implement our own parsing functions.

As I have demonstrated, if you want a table, it takes two lines to get it.

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: package.(c)path as tables

Sean Conner
In reply to this post by Dirk Laurie-2
It was thus said that the Great Dirk Laurie once stated:

> 2017-09-06 10:26 GMT+02:00 Daurnimator <[hidden email]>:
> > At the moment, package.path and package.cpath are strings containing
> > semicolon separated lists.
> > I'd like to propose that they are transitioned to be sequences to
> > allow for easier programmatic manipulation.
> >
> > The environment variables LUA_PATH, LUA_PATH_5_4, LUA_CPATH, etc
> > wouldn't change: they'd just need to be parsed when the package
> > library is loaded.
> >
> > In the next release we can allow package.path and package.cpath to be
> > strings for compatibility.
> > However the construct `package.path = package.path ..
> > ";/my/path/?.lua" would break and need to be changed to
> > `table.insert(package.path, "/my/path/?.lua")`
> > If this breakage is too much, perhaps we could have three phases:
> >   - phase 1. we allow table form, but still use string form on lua
> > command line application load.
> >   - phase 2. we change the default to be table form; string form is deprecated
> >   - phase 3. we remove support for string form.
>
> local path = {}
> package.path:gsub("[^;]*,function(x) path[#path+1]=x end

  Well, fix that up a bit:

local path = {}
package.path:gsub("[^;]*",function(x) path[#path+1]=x end)

Lua 5.1 returns:

path =
{
  [1] = "./?.lua",
  [2] = "",
  [3] = "/usr/local/share/lua/5.1/?.lua",
  [4] = "",
  [5] = "/usr/local/share/lua/5.1/?/init.lua",
  [6] = "",
  [7] = "/usr/local/lib/lua/5.1/?.lua",
  [8] = "",
  [9] = "/usr/local/lib/lua/5.1/?/init.lua",
  [10] = "",
}

while Lua 5.3 will return:

"path" =
{
  [1] = "/usr/local/share/lua/5.3/?.lua",
  [2] = "/usr/local/share/lua/5.3/?/init.lua",
  [3] = "/usr/local/lib/lua/5.3/?.lua",
  [4] = "/usr/local/lib/lua/5.3/?/init.lua",
  [5] = "./?.lua",
  [6] = "./?/init.lua",
}

Perhaps a better version:

local path = {}
package.path:gsub("[^;]+",function(x) path[#path+1]=x end)

Works better, but I'm still bothered by that ";", which *could* be changed
by an implementation.  Perhaps:

local path = {}
package.path:gsub(
        "[^%" .. package.config:sub(3,3) .. "]+",
        function(p) path[#path + 1] = p end
)

  -spc (There ... that's better)

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: package.(c)path as tables

Russell Haley
In reply to this post by Daurnimator
On Wed, Sep 6, 2017 at 1:26 AM, Daurnimator <[hidden email]> wrote:

> At the moment, package.path and package.cpath are strings containing
> semicolon separated lists.
> I'd like to propose that they are transitioned to be sequences to
> allow for easier programmatic manipulation.
>
> The environment variables LUA_PATH, LUA_PATH_5_4, LUA_CPATH, etc
> wouldn't change: they'd just need to be parsed when the package
> library is loaded.
>
> In the next release we can allow package.path and package.cpath to be
> strings for compatibility.
> However the construct `package.path = package.path ..
> ";/my/path/?.lua" would break and need to be changed to
> `table.insert(package.path, "/my/path/?.lua")`
> If this breakage is too much, perhaps we could have three phases:
>   - phase 1. we allow table form, but still use string form on lua
> command line application load.
>   - phase 2. we change the default to be table form; string form is deprecated
>   - phase 3. we remove support for string form.

I personally think parsing strings is for the birds (and Dirk and Sean
apparently. Thanks for those little nuggets) but can you provide some
concrete examples of where this would improve the language? Speed,
reliability, readability? Do you have a need to be adding and removing
many items to the path during runtime where this would be useful?

Cheers,

Russ

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: package.(c)path as tables

Dirk Laurie-2
2017-09-07 6:08 GMT+02:00 Russell Haley <[hidden email]>:

> I personally think parsing strings is for the birds (and Dirk and Sean
> apparently.

In most languages, yes, but Lua's pattern system is so convenient and
its string library is so short an sweet.

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: package.(c)path as tables

Russell Haley
In reply to this post by Dirk Laurie-2
On Wed, Sep 6, 2017 at 4:32 AM, Dirk Laurie <[hidden email]> wrote:

> 2017-09-06 10:26 GMT+02:00 Daurnimator <[hidden email]>:
>> At the moment, package.path and package.cpath are strings containing
>> semicolon separated lists.
>> I'd like to propose that they are transitioned to be sequences to
>> allow for easier programmatic manipulation.
>>
>> The environment variables LUA_PATH, LUA_PATH_5_4, LUA_CPATH, etc
>> wouldn't change: they'd just need to be parsed when the package
>> library is loaded.
>>
>> In the next release we can allow package.path and package.cpath to be
>> strings for compatibility.
>> However the construct `package.path = package.path ..
>> ";/my/path/?.lua" would break and need to be changed to
>> `table.insert(package.path, "/my/path/?.lua")`
>> If this breakage is too much, perhaps we could have three phases:
>>   - phase 1. we allow table form, but still use string form on lua
>> command line application load.
>>   - phase 2. we change the default to be table form; string form is deprecated
>>   - phase 3. we remove support for string form.
>
> local path = {}
> package.path:gsub("[^;]*,function(x) path[#path+1]=x end
>
> A breaking change to save having to code that? DMML

--you were missing a double quote and a parenthesis
local path = {}
package.path:gsub("[^;]*",function(x) path[#path+1]=x end)

--proof
for i,v in pairs(path) do
print(i,v)
end

Thanks for this,
Russ

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: package.(c)path as tables

Russell Haley
In reply to this post by Dirk Laurie-2
On Wed, Sep 6, 2017 at 9:50 PM, Dirk Laurie <[hidden email]> wrote:
> 2017-09-07 6:08 GMT+02:00 Russell Haley <[hidden email]>:
>
>> I personally think parsing strings is for the birds (and Dirk and Sean
>> apparently.
>
> In most languages, yes, but Lua's pattern system is so convenient and
> its string library is so short an sweet.

Agreed. String manipulation is a feature I often bring up when talking
with other developers about Lua (especially C developers).

Russ

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: package.(c)path as tables

Martin
In reply to this post by Daurnimator
I see two arguments in this thread versus proposal:

* "operating systems do it that way"
* string parsing is easy

Regarding first. Even if operating system stores search paths as
string, should Lua language export this behavior?

Second. Yes, string parsing is strong side of Lua. Should it
be considered the natural way to communicate?

For example why not add some magic char "results separator" "@" so
that "return 1, 2" will return string "1@2". Yes, it may be parsed
to table in two lines (two _correct_ lines).

I'd wish that language standard libraries use more Lua.

Ideally, to add path "../strange;name/*.lua" I'd wish to call

  package.add_search_path('../strange;name/*.lua')

and do not know about special characters from "package.config"
for name substitution and records delimiting.

Current proposal (store "package.path" as sequence of strings)
releases user from knowing about paths delimiter character.
So it simplifies code, making it more understandable for "common
programmer".

-- Martin

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: package.(c)path as tables

Martin
In reply to this post by nobody
On 09/07/2017 02:17 AM, nobody wrote:

> On 2017-09-06 20:03, Martin wrote:
>> To get terms (for example, when searching module "by hands") we have
>> to implement our own parsing functions.
>
> What did you need to do that `package.searchpath`[1] wasn't enough for?
>
> -- nobody
>
>
> [1]: https://www.lua.org/manual/5.3/manual.html#pdf-package.searchpath
>

You're right. I forgot about this function and implemented limited
version of searchpath() by hands.

-- Martin

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: package.(c)path as tables

Sean Conner
In reply to this post by Martin
It was thus said that the Great Martin once stated:

> Ideally, to add path "../strange;name/*.lua" I'd wish to call
>
>   package.add_search_path('../strange;name/*.lua')
>
> and do not know about special characters from "package.config"
> for name substitution and records delimiting.
>
> Current proposal (store "package.path" as sequence of strings)
> releases user from knowing about paths delimiter character.
> So it simplifies code, making it more understandable for "common
> programmer".

  But ... if you want to add "../strange;name/*.lua" to the path, you are
already dealing with path delimeter characters and substitution characters
(I assume that "*" means the same as "?", which is the character Lua uses
for module name substitution).  If anything, it might be nice to have:

        package.config =
        {
          directory      = "/",
          separator      = ";",
          substitution   = "?",
          exec_directory = "!",
          ignore         = "-"
        }

instead of what we have now.  But I digress ...

  -spc


Reply | Threaded
Open this post in threaded view
|

Re: Proposal: package.(c)path as tables

Dirk Laurie-2
In reply to this post by Martin
2017-09-07 10:29 GMT+02:00 Martin <[hidden email]>:
> I see two arguments in this thread versus proposal:
>
> * "operating systems do it that way"
> * string parsing is easy

You miss a most compelling third: it is a breaking change. It will
affect all programs that currently modify/change package.path.

For a breaking change, arguments of like "elegant", "more
understandable", "easier to manipulate" are not strong enough,
especially when a simple workaraound is available.

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: package.(c)path as tables

Martin
In reply to this post by Daurnimator
(I have troubles with my email client, so quoting manually.)

Dirk Laurie:
> You miss a most compelling third: it is a breaking change. It will
> affect all programs that currently modify/change package.path.

> For a breaking change, arguments of like "elegant", "more
> understandable", "easier to manipulate" are not strong enough,
> especially when a simple workaraound is available.

There is simple workaround for this too:

  v5.5 Introduce "package.path_t", "package.cpath_t".
       Declare "package.path", "package.cpath" deprecated.
  v6.0 Drop "package.path", "package.cpath"

will be enough. After it we'll get ugly names but code will look
better anyway (IMO).

Dropping "module()", "package.seeall()", moving "unpack()" - that was
breaking changes.

So there are precedents and a way to implement proposal without
instant hurting code at next version.

I see a better way to manage list of directories. But don't consider
this as argument versus proposal.

--[[
  Anyway even if it will be implemented it's not the final answer.

  In v6.1 someone will note that there is no duplicates checking and
  he must write "for" loop before table insert.

  What is really needed (IMO) is set-like data structure with
  two core operations: add, remove.

  (table: key - path (string), value - true).
]]

-- Martin

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: package.(c)path as tables

Dirk Laurie-2
2017-09-07 12:46 GMT+02:00 Martin <[hidden email]>:

> I see a better way to manage list of directories. But don't consider
> this as argument versus proposal.

Sorry, you just shot yourself in the foot.

The most glaring difficulty in Lua at present is that working with
directories requires either os.execute with OS-dependent calls
to change the working directory, or manipulation of filenames
(i.e. precisely the sort of string parsing that the proposal aims
to reduce).

What we need is some abstraction layer bring directories into
the io or os library. I.e. on a system that doesn't actually have
them, make it feel to the Lua programmer as if it has.

This is much more urgent than pottering about with package.path.
In fact, once we have this, a better way than the current proposal
of working with a directory list will probably be staring us in the face.

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: package.(c)path as tables

Neico
In reply to this post by Dirk Laurie-2
if type( package.config ) == "string" then
     local org_config = org_config or package.config

     local config = {};
     config.dirsep,
     config.pathsep,
     config.file_mark,
     config.execdir_mark,
     config.ignore_mark = table.unpack( org_config:split( "\n" ) ); --
split is one of those string funcs that lua really lacks, it's basically
a simple gmatch( "()" .. seperator .. "()" ) loop

     package.config = setmetatable( config, { ["__tostring"] = function(
tTable ) return org_config end } )
end

one time conversion, offers backwards compatibility (sort of) while
offering the new method, same technique can be applied to the paths
(this get's a bit easier if implemented natively instead)

The only drawback is that the string lib funcs don't check for
__tostring (neither does the colon syntax when applied to a table),
which is really a shame as it would give more weight to the metamethod

So in the end this won't work as well unless some other changes are done
along with it, but hey, at least it's there...


Reply | Threaded
Open this post in threaded view
|

Re: Proposal: package.(c)path as tables

Oliver Kroth
In reply to this post by Martin


Am 07.09.2017 um 12:46 schrieb Martin:
>> There is simple workaround for this too:
>>
>>    v5.5 Introduce "package.path_t", "package.cpath_t".
>>
not that I see a benefit of creating another interface for a more or
less static configuration item...
Why not name them package.paths and package.cpaths to indicate the
plurality and avoid this technical markup '_t' ?
And going further; it is possible to implement this in pure Lua using
metatables...

--
Oliver

12