Building luaffifb with MSVC

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Building luaffifb with MSVC

Dibyendu Majumdar
Hi,

I have been trying to build luaffifb
(https://github.com/facebookarchive/luaffifb) project using MSVC and
faced some issues. It seems that support for MSVC has fallen by the
wayside. Note also that the project is now archived :-(

My attempt is here: https://github.com/dibyendumajumdar/ravi-ffi

This will be part of Ravi Distro project.

I updated dynasm to the latest version. And I am using CMake, but I
have not yet tried building on Linux or Mac OSX.

The tests don't pass yet as the code that looks up DLLs for C standard
library does not seem to work.

If anyone has tried building luaffifb on Windows using MSVC then would
love to hear from you. I know that the original luaffi project is
included in LuaDist.

BTW I wanted to understand best practices for naming of shared
libraries in Lua - if there is a good set of standards one should
follow then please point me to them.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Building luaffifb with MSVC

Steven Johnson
Hi.

I did the changes discussed in the pull request there, though I didn't come away with any great insights as it was all rather mechanical. It
sounds like you might already be building on that though, unless your comments about tests just come from reading through some of the
discussion?

I seem to recall the C namespace just getting absorbed into another one rather than being outright missing. This sounds like it might be a
simple problem (wrong table index? maybe an off-by-one caused by some platform defines?) but I ended up getting distracted before looking
into it.

On Tue, Feb 6, 2018 at 1:10 PM, Dibyendu Majumdar <[hidden email]> wrote:
Hi,

I have been trying to build luaffifb
(https://github.com/facebookarchive/luaffifb) project using MSVC and
faced some issues. It seems that support for MSVC has fallen by the
wayside. Note also that the project is now archived :-(

My attempt is here: https://github.com/dibyendumajumdar/ravi-ffi

This will be part of Ravi Distro project.

I updated dynasm to the latest version. And I am using CMake, but I
have not yet tried building on Linux or Mac OSX.

The tests don't pass yet as the code that looks up DLLs for C standard
library does not seem to work.

If anyone has tried building luaffifb on Windows using MSVC then would
love to hear from you. I know that the original luaffi project is
included in LuaDist.

BTW I wanted to understand best practices for naming of shared
libraries in Lua - if there is a good set of standards one should
follow then please point me to them.

Regards
Dibyendu


Reply | Threaded
Open this post in threaded view
|

Re: Building luaffifb with MSVC

Dibyendu Majumdar
Hi Steven,

On 6 February 2018 at 20:06, Steven Johnson <[hidden email]> wrote:
> I did the changes discussed in the pull request there, though I didn't come
> away with any great insights as it was all rather mechanical. It
> sounds like you might already be building on that though, unless your
> comments about tests just come from reading through some of the
> discussion?
>

Unfortunately I didn't look into the pull request or the discussion -
so I was reporting my own findings. But I will look into the changes
you mention now that I am aware of it.

> I seem to recall the C namespace just getting absorbed into another one
> rather than being outright missing. This sounds like it might be a
> simple problem (wrong table index? maybe an off-by-one caused by some
> platform defines?) but I ended up getting distracted before looking
> into it.
>

I thought it might be that it isn't loading the right DLL. LuaJIT
appears to do it differently - it loads the MS C runtime by looking
for the library that contains a symbol - this seems like a good
approach to me.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Building luaffifb with MSVC

Dibyendu Majumdar
On 6 February 2018 at 20:11, Dibyendu Majumdar <[hidden email]> wrote:

>> I seem to recall the C namespace just getting absorbed into another one
>> rather than being outright missing. This sounds like it might be a
>> simple problem (wrong table index? maybe an off-by-one caused by some
>> platform defines?) but I ended up getting distracted before looking
>> into it.
>>
>
> I thought it might be that it isn't loading the right DLL. LuaJIT
> appears to do it differently - it loads the MS C runtime by looking
> for the library that contains a symbol - this seems like a good
> approach to me.
>

It seems the library is doing the same as LuaJIT - i.e. looking for a
DLL that contains the symbol _fmode, so I am mistaken about what the
problem might be.

Regards

Reply | Threaded
Open this post in threaded view
|

Re: Building luaffifb with MSVC

Dibyendu Majumdar
Steven Johnson wrote:

>>> I seem to recall the C namespace just getting absorbed into another one
>>> rather than being outright missing. This sounds like it might be a
>>> simple problem (wrong table index? maybe an off-by-one caused by some
>>> platform defines?) but I ended up getting distracted before looking
>>> into it.

I have made some progress with this.

First problem appears to be that we need an additional MSVC compiler
define - as implemented in LuaJIT
(https://github.com/LuaJIT/LuaJIT/commit/02b4b1e55633c36f370058e7601c77ba561e2c8a).
This has solved the issue of finding ffi.C.sprintf).

Secondly, on Windows a symbol loaded by ffi.load() does not go into
global namespace, whereas this works on Posix environment, but only
supposed to happen if a second argument is supplied to ffi.load - see
here (http://luajit.org/ext_ffi_api.html). There is possibly a bug in
luaffifb where symbols are perhaps being loaded into global namespace
when they should not be. This explains why on Windows you need to
access the library module in the dlls table to access functions in the
libtest module. Btw this appears to be a regression in luaffifb - i.e.
original code does not attempt to access symbols in libtest using
ffi.C.

Next issue is that on Windows <complex.h> is not ISO C compliant - and
the check for this was incorrect (actually also a regression from
luaffi where it is correct).

Fourth issue is the use of %p in some tests - unfortunately there is
no defined output format for %p, and on Windows this is different.
After fixing this I found an issue (possibly a bug) where float
arguments are not being passed correctly ... to be investigated.

Fixes are all in my repository: https://github.com/dibyendumajumdar/ravi-ffi

I cannot test all this on Linux right now as I am working remotely ,
but on Mac OSX the tests passed so I assume they would also pass on
Linux.

Windows version is failing towards the end due to the potential bug
with passing float arguments.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Building luaffifb with MSVC

Dibyendu Majumdar
In reply to this post by Dibyendu Majumdar
On 13 February 2018 at 18:47, Dibyendu Majumdar <[hidden email]> wrote:

> Steven Johnson wrote:
>
>>>> I seem to recall the C namespace just getting absorbed into another one
>>>> rather than being outright missing. This sounds like it might be a
>>>> simple problem (wrong table index? maybe an off-by-one caused by some
>>>> platform defines?) but I ended up getting distracted before looking
>>>> into it.
>
> I have made some progress with this.
>
> First problem appears to be that we need an additional MSVC compiler
> define - as implemented in LuaJIT
> (https://github.com/LuaJIT/LuaJIT/commit/02b4b1e55633c36f370058e7601c77ba561e2c8a).
> This has solved the issue of finding ffi.C.sprintf).
>
> Secondly, on Windows a symbol loaded by ffi.load() does not go into
> global namespace, whereas this works on Posix environment, but only
> supposed to happen if a second argument is supplied to ffi.load - see
> here (http://luajit.org/ext_ffi_api.html). There is possibly a bug in
> luaffifb where symbols are perhaps being loaded into global namespace
> when they should not be. This explains why on Windows you need to
> access the library module in the dlls table to access functions in the
> libtest module. Btw this appears to be a regression in luaffifb - i.e.
> original code does not attempt to access symbols in libtest using
> ffi.C.
>
> Next issue is that on Windows <complex.h> is not ISO C compliant - and
> the check for this was incorrect (actually also a regression from
> luaffi where it is correct).
>
> Fourth issue is the use of %p in some tests - unfortunately there is
> no defined output format for %p, and on Windows this is different.
> After fixing this I found an issue (possibly a bug) where float
> arguments are not being passed correctly ... to be investigated.
>

Looks like there was a bug in luaffi when passing parameters to
functions on the stack on x86-64 platform. I have not found definitive
documentation clarifying this (if anyone can provide a reference that
would be very helpful) but it seems that stack arguments must be at 8
byte boundaries. It also appears that the Facebook fork applied a
partial fix (covering 1 of out 4 scenarios) - so the bug is present in
luaffifb as well. Anyway I have applied a fix and now the tests pass
on Windows. I would have expected the impacted tests to fail on all
X86-64 platforms so not sure yet why they were passing on Mac OSX.

As above is quite a serious bug, I need to beef up the tests in this
module to ensure that as many scenarios are tested as possible.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Building luaffifb with MSVC

Thomas Buergel
> Looks like there was a bug in luaffi when passing parameters to
> functions on the stack on x86-64 platform. I have not found definitive
> documentation clarifying this (if anyone can provide a reference that
> would be very helpful) but it seems that stack arguments must be at 8
> byte boundaries.

The two dominant x86-64 calling conventions in an overview:
https://en.wikipedia.org/wiki/X86_calling_conventions#x86-64_calling_conventions

For once, the one used on Windows is simpler:
https://docs.microsoft.com/en-us/cpp/build/calling-convention

Especially varargs can be a challenge on the SysV x86-64 ABI:
https://blog.nelhage.com/2010/10/amd64-and-va_arg/

Reply | Threaded
Open this post in threaded view
|

Re: Building luaffifb with MSVC

Dibyendu Majumdar
On 15 February 2018 at 06:40, Thomas Buergel <[hidden email]> wrote:

>> Looks like there was a bug in luaffi when passing parameters to
>> functions on the stack on x86-64 platform. I have not found definitive
>> documentation clarifying this (if anyone can provide a reference that
>> would be very helpful) but it seems that stack arguments must be at 8
>> byte boundaries.
>
> The two dominant x86-64 calling conventions in an overview:
> https://en.wikipedia.org/wiki/X86_calling_conventions#x86-64_calling_conventions
>
> For once, the one used on Windows is simpler:
> https://docs.microsoft.com/en-us/cpp/build/calling-convention
>
> Especially varargs can be a challenge on the SysV x86-64 ABI:
> https://blog.nelhage.com/2010/10/amd64-and-va_arg/
>


Thank you - I was aware of the first two references but there doesn't
seem to be a discussion on the requirements of size/alignment of
arguments passed on the stack.

I found following:

https://stackoverflow.com/questions/32614007/size-and-alignment-of-x64-stack-arguments

And:

http://refspecs.linuxfoundation.org/elf/x86_64-abi-0.99.pdf

Above says:

The size of each argument gets rounded up to eightbytes

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Building luaffifb with MSVC

KHMan
On 2/15/2018 3:01 PM, Dibyendu Majumdar wrote:

> On 15 February 2018 at 06:40, Thomas Buergel wrote:
>>> Looks like there was a bug in luaffi when passing parameters to
>>> functions on the stack on x86-64 platform. I have not found definitive
>>> documentation clarifying this (if anyone can provide a reference that
>>> would be very helpful) but it seems that stack arguments must be at 8
>>> byte boundaries.
>>
>> The two dominant x86-64 calling conventions in an overview:
>> https://en.wikipedia.org/wiki/X86_calling_conventions#x86-64_calling_conventions
>>
>> For once, the one used on Windows is simpler:
>> https://docs.microsoft.com/en-us/cpp/build/calling-convention
>>
>> Especially varargs can be a challenge on the SysV x86-64 ABI:
>> https://blog.nelhage.com/2010/10/amd64-and-va_arg/
>>
>
>
> Thank you - I was aware of the first two references but there doesn't
> seem to be a discussion on the requirements of size/alignment of
> arguments passed on the stack.
>
> I found following:
>
> https://stackoverflow.com/questions/32614007/size-and-alignment-of-x64-stack-arguments
>
> And:
>
> http://refspecs.linuxfoundation.org/elf/x86_64-abi-0.99.pdf
>
> Above says:
>
> The size of each argument gets rounded up to eightbytes

Don't forget Agner Fog (I guess he's more familiar to older
folks), see the doc on calling conventions:

http://www.agner.org/optimize/


--
Cheers,
Kein-Hong Man (esq.)
Selangor, Malaysia


Reply | Threaded
Open this post in threaded view
|

Re: Building luaffifb with MSVC

Dibyendu Majumdar
On 15 February 2018 at 07:33, KHMan <[hidden email]> wrote:

> On 2/15/2018 3:01 PM, Dibyendu Majumdar wrote:
>>
>> On 15 February 2018 at 06:40, Thomas Buergel wrote:
>>>>
>>>> Looks like there was a bug in luaffi when passing parameters to
>>>> functions on the stack on x86-64 platform. I have not found definitive
>>>> documentation clarifying this (if anyone can provide a reference that
>>>> would be very helpful) but it seems that stack arguments must be at 8
>>>> byte boundaries.
>>>
>>>
>>> The two dominant x86-64 calling conventions in an overview:
>>>
>>> https://en.wikipedia.org/wiki/X86_calling_conventions#x86-64_calling_conventions
>>>
>>> For once, the one used on Windows is simpler:
>>> https://docs.microsoft.com/en-us/cpp/build/calling-convention
>>>
>>> Especially varargs can be a challenge on the SysV x86-64 ABI:
>>> https://blog.nelhage.com/2010/10/amd64-and-va_arg/
>>>
>>
>>
>> Thank you - I was aware of the first two references but there doesn't
>> seem to be a discussion on the requirements of size/alignment of
>> arguments passed on the stack.
>>
>> I found following:
>>
>>
>> https://stackoverflow.com/questions/32614007/size-and-alignment-of-x64-stack-arguments
>>
>> And:
>>
>> http://refspecs.linuxfoundation.org/elf/x86_64-abi-0.99.pdf
>>
>> Above says:
>>
>> The size of each argument gets rounded up to eightbytes
>
>
> Don't forget Agner Fog (I guess he's more familiar to older folks), see the
> doc on calling conventions:
>
> http://www.agner.org/optimize/
>

Thank you, the calling conventions document appears quite detailed. It
confirms above, as:

The 64 bit systems keep the stack aligned by 16. The stack word size
is 8 bytes, but the stack must be aligned by 16 before any call
instruction.

Each parameter must take a whole number of stack entries. If a
parameter is smaller than the stack word size then the rest of that
stack entry is unused.


Regards