5.4.0 Makefile patch for FreeBSD Was: Re: [ANN] Lua 5.4.0 (work1) now available

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

5.4.0 Makefile patch for FreeBSD Was: Re: [ANN] Lua 5.4.0 (work1) now available

Russell Haley


On Wed, Mar 14, 2018 at 12:38 AM, Andrew Gierth <[hidden email]> wrote:
>>>>> "Luiz" == Luiz Henrique de Figueiredo <[hidden email]> writes:

 Luiz> All feedback welcome. Thanks.

Building on freebsd doesn't work without adding CC=cc (since gcc is not
installed by default and if you install it from ports, it's usually
under a version-specific name such as gcc6), and also enabling readline
requires adding -I/usr/local/include and -L/usr/local/lib, or tweaking
things to use libedit instead.

I've tried these, which all seem to work on my system (though obviously
the presence of a port build would make this somewhat moot):

# with base system libedit:
freebsd:
        $(MAKE) $(ALL) CC="cc" SYSCFLAGS="-DLUA_USE_LINUX -DLUA_USE_READLINE -I/usr/include/edit" SYSLIBS="-Wl,-E -ledit"

# ... with readline from ports:
freebsd-readline:
        $(MAKE) $(ALL) CC="cc" SYSCFLAGS="-DLUA_USE_LINUX -DLUA_USE_READLINE -I/usr/local/include" SYSLIBS="-Wl,-E -L/usr/local/lib -lreadline"

# ... with editing disabled
freebsd-noedit:
        $(MAKE) $(ALL) CC="cc" SYSCFLAGS="-DLUA_USE_LINUX" SYSLIBS="-Wl,-E"

and speaking of readline, it would still be good for lua.c to set
rl_readline_name and (maybe) rl_inhibit_completion as discussed
recently.

Hi,

I'd be really pleased to see the Makefile patched for FreeBSD, and all the BSDs for 5.4. There were a few discussions last summer about what's actually required to build on FreeBSD, other BSDs, Mac and Solaris. I'd like to help out with the other BSDs, but I'm mostly interested in FreeBSD and Windows right now. On FreeBSD, an official patch would be great, but I could live with patching it myself in ports. Some items of interest to me:

- As Andrew nicely illustrated, libeditline is the standard readline package on FreeBSD.

- Andrew: luaconf.h indicates to me that LUA_USE_READLINE is not required when using LUA_USE_LINUX?  Also, -fPIC is required for Arm and was added to x86/amd64. Do you have any input on -fPIC? I understood why some time ago, but my understanding was pretty weak.

- There was a lengthy discussion (on my part. tee hee) which concluded that dlopen library (or whatever it's called) is only required on glibc. My less than stellar memory says that I concluded (potentially) LUA_USE_DLOPEN isn't needed on ANY of the BSDs or OSX. I also checked what Solaris documentation and found that dlopen was also not required there (not surprisingly).

Given the above, for a 5.4.0 port, I'd like to see something as close as possible to this:
CC=cc
...
$(MAKE) $(ALL) SYSCFLAGS="-DLUA_USE_POSIX -DLUA_USE_READLINE -fPIC -I/usr/include/edit" SYSLIBS="-Wl,-E -ledit"

OR

Perhaps the Makefile can be modified to have a single target (that FreeBSD, BSD, MACOSX point to) using a new define of LUA_USE_NON_GNU? That define would then be  LUA_USE_POSIX and LUA_USE_READLINE?

There are other combinations that could be better I'm sure so I'm asking if there is stomach for that kind of change?

I tested much of this this some time ago but will retest in case there is interest. I've put a message on Google+ for the openbsd group and will send some emails for input from others.

Thanks for listening.

Russ



Thoughts?

Russ


--
Andrew.


Reply | Threaded
Open this post in threaded view
|

Re: 5.4.0 Makefile patch for FreeBSD

Andrew Gierth
>>>>> "Russell" == Russell Haley <[hidden email]> writes:

 Russell> - Andrew: luaconf.h indicates to me that LUA_USE_READLINE is
 Russell> not required when using LUA_USE_LINUX?

This seems to have changed from 5.3 to 5.4 - 5.3's luaconf.h defines
LUA_USE_READLINE, but 5.4's does not.

 Russell> Also, -fPIC is required for Arm

-fPIC is required to build a shared object, which the distributed
makefiles do not do. Nothing to do with arm vs. intel; the build works
fine without -fPIC on arm (at least on 32-bit arm, which is all I
currently have available).

The port build builds a shared library, so it has to add -fPIC for that
to work.

 Russell> - There was a lengthy discussion (on my part. tee hee) which
 Russell> concluded that dlopen library (or whatever it's called) is
 Russell> only required on glibc. My less than stellar memory says that
 Russell> I concluded (potentially) LUA_USE_DLOPEN isn't needed on ANY
 Russell> of the BSDs or OSX.

This is wrong. LUA_USE_DLOPEN is what tells Lua that dlopen() and
dlsym() are available for dynamic loading; if you don't define it, then
you won't be able to require C modules. (The only alternate loading
scheme supported is LUA_DL_DLL for Windows.)

What's different between freebsd and linux here is that linux requires
-ldl at link time and freebsd does not.

--
Andrew.

Reply | Threaded
Open this post in threaded view
|

Re: 5.4.0 Makefile patch for FreeBSD

Dirk Laurie-2
2018-03-15 8:58 GMT+02:00 Andrew Gierth <[hidden email]>:

> This seems to have changed from 5.3 to 5.4 - 5.3's luaconf.h defines
> LUA_USE_READLINE, but 5.4's does not.

5.4 has separate linux-readline and linux-noreadline make targets. 5.3 did not.

Reply | Threaded
Open this post in threaded view
|

Re: 5.4.0 Makefile patch for FreeBSD Was: Re: [ANN] Lua 5.4.0 (work1) now available

William Ahern
In reply to this post by Russell Haley
On Wed, Mar 14, 2018 at 11:15:35PM -0700, Russell Haley wrote:

> On Wed, Mar 14, 2018 at 12:38 AM, Andrew Gierth <[hidden email]
> > wrote:
>
> > >>>>> "Luiz" == Luiz Henrique de Figueiredo <[hidden email]>
> > writes:
> >
> >  Luiz> All feedback welcome. Thanks.
> >
> > Building on freebsd doesn't work without adding CC=cc (since gcc is not
> > installed by default and if you install it from ports, it's usually
> > under a version-specific name such as gcc6), and also enabling readline
> > requires adding -I/usr/local/include and -L/usr/local/lib, or tweaking
> > things to use libedit instead.
> >
> > I've tried these, which all seem to work on my system (though obviously
> > the presence of a port build would make this somewhat moot):
> >
> > # with base system libedit:
> > freebsd:
> >         $(MAKE) $(ALL) CC="cc" SYSCFLAGS="-DLUA_USE_LINUX
> > -DLUA_USE_READLINE -I/usr/include/edit" SYSLIBS="-Wl,-E -ledit"
> >
> > # ... with readline from ports:
> > freebsd-readline:
> >         $(MAKE) $(ALL) CC="cc" SYSCFLAGS="-DLUA_USE_LINUX
> > -DLUA_USE_READLINE -I/usr/local/include" SYSLIBS="-Wl,-E -L/usr/local/lib
> > -lreadline"
> >
> > # ... with editing disabled
> > freebsd-noedit:
> >         $(MAKE) $(ALL) CC="cc" SYSCFLAGS="-DLUA_USE_LINUX" SYSLIBS="-Wl,-E"
> >
> > and speaking of readline, it would still be good for lua.c to set
> > rl_readline_name and (maybe) rl_inhibit_completion as discussed
> > recently.
> >
> > Hi,
>
> I'd be really pleased to see the Makefile patched for FreeBSD, and all the
> BSDs for 5.4. There were a few discussions last summer about what's
> actually required to build on FreeBSD, other BSDs, Mac and Solaris. I'd
> like to help out with the other BSDs, but I'm mostly interested in FreeBSD
> and Windows right now. On FreeBSD, an official patch would be great, but I
> could live with patching it myself in ports. Some items of interest to me:
>
> - As Andrew nicely illustrated, libeditline is the standard readline
> package on FreeBSD.
>
> - Andrew: luaconf.h indicates to me that LUA_USE_READLINE is not required
> when using LUA_USE_LINUX?  Also, -fPIC is required for Arm and was added to
> x86/amd64. Do you have any input on -fPIC? I understood why some time ago,
> but my understanding was pretty weak.
>
> - There was a lengthy discussion (on my part. tee hee) which concluded that
> dlopen library (or whatever it's called) is only required on glibc. My less
> than stellar memory says that I concluded (potentially) LUA_USE_DLOPEN
> isn't needed on ANY of the BSDs or OSX. I also checked what Solaris
> documentation and found that dlopen was also not required there (not
> surprisingly).
>
> Given the above, for a 5.4.0 port, I'd like to see something as close as
> possible to this:
> CC=cc
> ...
> $(MAKE) $(ALL) SYSCFLAGS="-DLUA_USE_POSIX -DLUA_USE_READLINE -fPIC
> -I/usr/include/edit" SYSLIBS="-Wl,-E -ledit"
>
> OR
>
> Perhaps the Makefile can be modified to have a single target (that FreeBSD,
> BSD, MACOSX point to) using a new define of LUA_USE_NON_GNU? That define
> would then be  LUA_USE_POSIX and LUA_USE_READLINE?
>
> There are other combinations that could be better I'm sure so I'm asking if
> there is stomach for that kind of change?

[For the specific problem of overriding these variables jump to bottom. I
originally replied to respond with an alternative naming scheme that more
directly and more idiomatically addresses porting requirements, but now it's
too late to rearrange everything.]

There's a systematic and consistent way to make specifying and changing
these parameters simple for porters, expounding upon the autotools model and
the GNU style guide--both what they prescribe (which is much less than the
following) and how they're used in practice (falls just short of the
following). Using the above two dependencies as an examples

  READLINE_CPPFLAGS= # flags to locate readline headers + dependencies
  READLINE_CFLAGS=   # flags for buildings object linkable to readline
  READLINE_LDFLAGS=  # flags to locate readline library + dependencies
  READLINE_LDLIBS=   # flags to link readline + depdendecnies

  DL_CPPFLAGS=
  DL_CFLAGS=
  DL_LDFLAGS=
  DL_LDLIBS=

However you provide convenient defaults, by making each of them
independently specifiable as feature groups _and_ by compilation phase it
becomes trivial for porters to control a build without modifying the
Makefile in most common situations. (Whether porters recognize and
appreciate this ability before hacking and patching the Makefile is another
matter.) Some of the prefixes could be left out, especially for particular
features or behaviors, but I try not to deviate from the system as the whole
point is consistency and flexibility without foreseeing specific use cases.

In autotools projects the above feature-group flags are always consolidated
like so,

  ...
  CPPFLAGS =   # default application flags easily overrideable
  MYCPPFLAGS = # so you're not forced to clobber CPPFLAGS just to add a flag
  ALL_CPPFLAGS = $(READLINE_CPPFLAGS) $(DL_CPPFLAGS) $(CPPFLAGS)
                 $(MYCPPFLAGS)

This is all very verbose and repetitive, but it's _simple_, _consistent_,
_readable_, and most importantly immediately _understandable_ both by
porters and experienced unix programmers. IME there's much more cognitive
burden and time wasted deliberating how and where to avoid the repetition
than habitually following the pattern.

While not as prevalent, the most common variable to use for independently
specifying flags for generating shared libraries is SOFLAGS, which defaults
to -shared for everything but macOS.

  SOFLAGS=-shared # or -bundle -undefined dynamic_lookup for macOS

(SOFLAGS can also be used to specify ELF sonames. Some projects break out
SONAME as a separate variable and default SOFLAGS to "-shared
-Wl,-soname,$(SONAME)", but for a project like Lua this is probably just
past the point where the costs exceed the benefits, especially if LDFLAGS
and MYLDFLAGS are available for injecting ad hoc flags without clobbering
other defaults.)

You can also extrapolate the scheme to make it easier to control aspects of
how internal objects are built. So, for example, I'll often have

  PIC_CPPFLAGS =
  PIC_CFLAGS = -fPIC # -xcode=pic13 on Solaris, crazy stuff for AIX
  PIC_LDFLAGS =
  PIC_LDLIBS =

  WARN_CPPFLAGS =
  WARN_CFLAGS = -Wall -fsanitize=address
  WARN_LDFLAGS =
  WARN_LDLIBS =

to make it easy to change specific sets of flags without having to
copy+paste all the others. (Similar to the logic behind having, e.g., CFLAGS
and MYCFLAGS). The above names naturally suggest themselves. I began using
them before I learned that they are actually relatively common. No
coincidence and strong evidence that this naming scheme puts us on the right
track.

The WARN case is also a great example of why you might want to preserve the
ability to control all phases independently even if you can't conceive of a
situation where, e.g., WARN_LDFLAGS would be useful. On systems where a GCC
or clang compiler with sanitizer support must be installed manually
(separate from the native compiler) you may need to explicitly specify
linker paths and libraries for the sanitizer runtime. (IIRC, this was at
least true in the early days of those features.) The amount of pain even one
user facing such a problem might save dwarfs the time spent writing and
maintaining the additional flags.

With any abstraction it's easy to go off the rails and end up with an
overwrought solution. But I think in it's basic form it's pragmatic and
agreeable.

The biggest headache with the existing Lua build, though, is that there's no
way to override the default, per-platform flag sets without forgoing the
pre-defined flags altogether. There's no way for me to say, "build for
macosx, _except_ use this flag instead of the default macosx flag". The
least intrusive way to fix this this is, from the platform target rules, to
specify the default flags as environment variables and then use the standard
(POSIX-defined) -e option. Make is defined by POSIX to propogate command
line options and macro definitions to recursive make invocations by passing
them through the MAKEFLAGS environment variable (whether or not the rule
uses the $(MAKE) macro; make always exports SHELL and MAKEFLAGS by default).

For example, I can override the definition of CC when using the macosx
target by applying this diff,

-   $(MAKE) $(ALL) SYSCFLAGS="-DLUA_USE_MACOSX" SYSLIBS="-lreadline" CC=cc
+   CC=cc $(MAKE) $(ALL) SYSCFLAGS="-DLUA_USE_MACOSX" SYSLIBS="-lreadline"

and then doing `make MAKE="make -e" macosx CC=gcc`. A proper fix would
look something like this (untested)

-   $(MAKE) $(ALL) SYSCFLAGS="-DLUA_USE_MACOSX" SYSLIBS="-lreadline" CC=cc
+   CC=cc SYSCFLAGS="-DLUA_USE_MACOSX" SYSLIBS="-lreadline" $(MAKE_E) $(ALL)

I haven't actually used the -e option before in real production builds, but
it works as expected on GNU make 3.81 (macOS) and OpenBSD make 6.2. At the
moment I can't test NetBSD make (aka bmake aka pmake, also used by FreeBSD)
or Solaris' default make, but I'd surprised if it didn't work correctly
there, either.

Now that I think about it, with the -e option I think I can finally adopt
Lua's clever method for conveniently aggregating default platform flags
using simple, portable constructs that don't confuse people (myself
included).

> I tested much of this this some time ago but will retest in case there is
> interest. I've put a message on Google+ for the openbsd group and will send
> some emails for input from others.
>
> Thanks for listening.
>
> Russ
>
>
>
> Thoughts?
>
> Russ
>
>
> --
> > Andrew.
> >
> >

Reply | Threaded
Open this post in threaded view
|

Re: 5.4.0 Makefile patch for FreeBSD Was: Re: [ANN] Lua 5.4.0 (work1) now available

William Ahern
On Thu, Mar 15, 2018 at 02:01:07AM -0700, William Ahern wrote:
<snip>

>
> The biggest headache with the existing Lua build, though, is that there's no
> way to override the default, per-platform flag sets without forgoing the
> pre-defined flags altogether. There's no way for me to say, "build for
> macosx, _except_ use this flag instead of the default macosx flag". The
> least intrusive way to fix this this is, from the platform target rules, to
> specify the default flags as environment variables and then use the standard
> (POSIX-defined) -e option. Make is defined by POSIX to propogate command
> line options and macro definitions to recursive make invocations by passing
> them through the MAKEFLAGS environment variable (whether or not the rule
> uses the $(MAKE) macro; make always exports SHELL and MAKEFLAGS by default).

I left out the pertinent aspect of -e: it changes the order of precedence so
that environment variables override variables defined in the makefile, but
still have lower precedence that those defined on the command-line or
inherited via MAKEFLAGS. The default precedence order is (quoting POSIX)

  Macro definitions shall be taken from the following sources, in the
  following logical order, before the makefile(s) are read.

  1. Macros specified on the make utility command line, in the order
  specified on the command line. It is unspecified whether the internal
  macros defined in Internal Macros are accepted from this source.

  2. Macros defined by the MAKEFLAGS environment variable, in the order
  specified in the environment variable. It is unspecified whether the
  internal macros defined in Internal Macros are accepted from this source.

  3. The contents of the environment, excluding the MAKEFLAGS and SHELL
  variables and including the variables with null values.

  4. Macros defined in the inference rules built into make.

  Macro definitions from these sources shall not override macro definitions
  from a lower-numbered source. Macro definitions from a single source (for
  example, the make utility command line, the MAKEFLAGS environment
  variable, or the other environment variables) shall override previous
  macro definitions from the same source.

 Macros defined in the makefile(s) shall override macro definitions that
 occur before them in the makefile(s) and macro definitions from source 4.
 If the -e option is not specified, macros defined in the makefile(s) shall
 override macro definitions from source 3. Macros defined in the makefile(s)
 shall not override macro definitions from source 1 or source 2.

Even Microsoft NMake seems to follow the same rules,

 The following list shows the order of precedence, from highest to lowest:

 1. A macro defined on the command line

 2. A macro defined in a makefile or include file

 3. An inherited environment-variable macro

 4. A macro defined in the Tools.ini file

 5. A predefined macro, such as CC and AS

 Use /E to cause macros inherited from environment variables to override
 makefile macros with the same name.

(https://docs.microsoft.com/en-us/cpp/build/precedence-in-macro-definitions)

> For example, I can override the definition of CC when using the macosx
> target by applying this diff,
>
> -   $(MAKE) $(ALL) SYSCFLAGS="-DLUA_USE_MACOSX" SYSLIBS="-lreadline" CC=cc
> +   CC=cc $(MAKE) $(ALL) SYSCFLAGS="-DLUA_USE_MACOSX" SYSLIBS="-lreadline"
<snip>


Reply | Threaded
Open this post in threaded view
|

Re: 5.4.0 Makefile patch for FreeBSD

Russell Haley
In reply to this post by Dirk Laurie-2
Thanks Dirk, Andrew, everyone.

https://reviews.freebsd.org/D14709

Need sleep.

Russ

On Thu, Mar 15, 2018 at 1:22 AM, Dirk Laurie <[hidden email]> wrote:
2018-03-15 8:58 GMT+02:00 Andrew Gierth <[hidden email]>:

> This seems to have changed from 5.3 to 5.4 - 5.3's luaconf.h defines
> LUA_USE_READLINE, but 5.4's does not.

5.4 has separate linux-readline and linux-noreadline make targets. 5.3 did not.


Reply | Threaded
Open this post in threaded view
|

Re: 5.4.0 Makefile patch for FreeBSD

Russell Haley


On Thu, Mar 15, 2018 at 11:30 PM, Russell Haley <[hidden email]> wrote:
Thanks Dirk, Andrew, everyone.

https://reviews.freebsd.org/D14709

Need sleep.

Russ

So far the only modification I've had to make to the original src/Makefile is to remove -lreadline

Russ 
On Thu, Mar 15, 2018 at 1:22 AM, Dirk Laurie <[hidden email]> wrote:
2018-03-15 8:58 GMT+02:00 Andrew Gierth <[hidden email]>:

> This seems to have changed from 5.3 to 5.4 - 5.3's luaconf.h defines
> LUA_USE_READLINE, but 5.4's does not.

5.4 has separate linux-readline and linux-noreadline make targets. 5.3 did not.



Reply | Threaded
Open this post in threaded view
|

Re: 5.4.0 Makefile patch for FreeBSD

Luiz Henrique de Figueiredo
> So far the only modification I've had to make to the original src/Makefile
> is to remove -lreadline

I'm confused. Someone else suggest here recently that the correct make
line for freebsd is

freebsd:
        $(MAKE) $(ALL) SYSCFLAGS="-DLUA_USE_LINUX -DLUA_USE_READLINE
-I/usr/include/edit" SYSLIBS="-Wl,-E -ledit" CC="cc"

Reply | Threaded
Open this post in threaded view
|

Re: 5.4.0 Makefile patch for FreeBSD

Andrew Gierth
>>>>> "Luiz" == Luiz Henrique de Figueiredo <[hidden email]> writes:

 >> So far the only modification I've had to make to the original
 >> src/Makefile is to remove -lreadline

 Luiz> I'm confused. Someone else suggest here recently that the correct
 Luiz> make line for freebsd is

 Luiz> freebsd:
 Luiz>         $(MAKE) $(ALL) SYSCFLAGS="-DLUA_USE_LINUX -DLUA_USE_READLINE
 Luiz> -I/usr/include/edit" SYSLIBS="-Wl,-E -ledit" CC="cc"

Russell's suggested port build is passing in MYCFLAGS and MYLDFLAGS
(incorrectly, should be using MYLIBS) to set the -I/usr/include/edit and
-ledit.

--
Andrew.

Reply | Threaded
Open this post in threaded view
|

Re: 5.4.0 Makefile patch for FreeBSD

Luiz Henrique de Figueiredo
>  Luiz> freebsd:
>  Luiz>         $(MAKE) $(ALL) SYSCFLAGS="-DLUA_USE_LINUX -DLUA_USE_READLINE
>  Luiz> -I/usr/include/edit" SYSLIBS="-Wl,-E -ledit" CC="cc"
>
> Russell's suggested port build is passing in MYCFLAGS and MYLDFLAGS
> (incorrectly, should be using MYLIBS) to set the -I/usr/include/edit and
> -ledit.

But the make line above works fine, right?

Reply | Threaded
Open this post in threaded view
|

Re: 5.4.0 Makefile patch for FreeBSD

Andrew Gierth
>>>>> "Luiz" == Luiz Henrique de Figueiredo <[hidden email]> writes:

 Luiz> freebsd:
 Luiz> $(MAKE) $(ALL) SYSCFLAGS="-DLUA_USE_LINUX -DLUA_USE_READLINE
 Luiz> -I/usr/include/edit" SYSLIBS="-Wl,-E -ledit" CC="cc"

 >> Russell's suggested port build is passing in MYCFLAGS and MYLDFLAGS
 >> (incorrectly, should be using MYLIBS) to set the -I/usr/include/edit
 >> and -ledit.

 Luiz> But the make line above works fine, right?

Right, for the purposes of being able to do "make freebsd" on the
distributed sources to get a static .a and executables. (The port build
does more than this, since it also builds a shared library.)

--
Andrew.

Reply | Threaded
Open this post in threaded view
|

Re: 5.4.0 Makefile patch for FreeBSD

Russell Haley


On Fri, Mar 16, 2018 at 5:46 AM, Andrew Gierth <[hidden email]> wrote:
>>>>> "Luiz" == Luiz Henrique de Figueiredo <[hidden email]> writes:

 Luiz> freebsd:
 Luiz> $(MAKE) $(ALL) SYSCFLAGS="-DLUA_USE_LINUX -DLUA_USE_READLINE
 Luiz> -I/usr/include/edit" SYSLIBS="-Wl,-E -ledit" CC="cc"

 >> Russell's suggested port build is passing in MYCFLAGS and MYLDFLAGS
 >> (incorrectly, should be using MYLIBS) to set the -I/usr/include/edit
 >> and -ledit.

 Luiz> But the make line above works fine, right?

Right, for the purposes of being able to do "make freebsd" on the
distributed sources to get a static .a and executables. (The port build
does more than this, since it also builds a shared library.)

Sorry I was unclear. My patch for the FreeBSD port files are now updated to correctly build Lua54 for FreeBSD including the shared object. It's currently based on work1. If the target as you specified is included in the Lua src/Makefile, I will update the port accordingly.

Cheers,

Russ 

--
Andrew.