Ho to trim file in lua?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Ho to trim file in lua?

Sergey Kovalev
How to trim existing file from 2Gb to 1Gb using lua io?

Reply | Threaded
Open this post in threaded view
|

Re: Ho to trim file in lua?

Paul K-2
> How to trim existing file from 2Gb to 1Gb using lua io?

Lua I/O doesn't provide any function to truncate a file. I supported
suggestion to add one. See this thread for a previous discussion on
this topic: http://lua.2524044.n2.nabble.com/Function-to-truncate-a-file-td7684482.html

If you need a portable solution, you can use Lua/APR
(http://peterodding.com/code/lua/apr/docs/#file:truncate) or fs
library (https://luapower.com/fs; requires FFI).

Paul.

On Thu, Jun 20, 2019 at 4:21 PM Sergey Kovalev <[hidden email]> wrote:
>
> How to trim existing file from 2Gb to 1Gb using lua io?
>

Reply | Threaded
Open this post in threaded view
|

Re: Ho to trim file in lua?

Sergey Kovalev
-- preallocate 1Gb
f=io.open("dummy.bin","wb") or error()
f:seek("set",1024*1024*1024-1)
f:write("1")
f:close()

пт, 21 июн. 2019 г. в 21:14, Philippe Verdy <[hidden email]>:

>
> May be some native extension can add this functionality to existing file objects (by adding i/o methods, or a way to pass native I/O requests via some escape mechanism that the extension will check itself.
>
> There are other features missing infile I/O, notably the possibility of prealocating empty space (without really having to write it on disk: it can jsut be marked in the filesystem allocation table has being allocated, but also implictly full of zeroes, freeing the OS from having to make physical reads as well, and it is very useful as well for "virtual" files. Another possibility is to virtualize a file's version (with separate commits of sectors/clusters into versioned branches), or the possibility of virtualizing multiple distinct files to share the same physical space by default, also virtualized when one file is written but not the other).
>
> Various filesystems have already implemented these, notably to better support virtual machines with lower cost and improved performance by reduced physical I/O; it is already the case on Linux (with recent versions of Ext4, including for Android devices) and Windows (with NTFS), and most probably as well on MacOS. And any OS that has now support for efficient implementation of virtual machines have added such support in at least one of their supported filesystems, or have added one which can also be used for other applications (including for installing the OS itself and maintaining it securely with "instant snashots").
>
> I wonder why POSIX did not try to extend the minimum set of file I/O API that a decent OS should support (or that can probably be easily emulated by small plugins in the implementation of virtual machines and scripting engines like Lua, even if sometimes it may be much slower than when using the native filesystem support).
>
>
> Le ven. 21 juin 2019 à 01:37, Paul K <[hidden email]> a écrit :
>>
>> > How to trim existing file from 2Gb to 1Gb using lua io?
>>
>> Lua I/O doesn't provide any function to truncate a file. I supported
>> suggestion to add one. See this thread for a previous discussion on
>> this topic: http://lua.2524044.n2.nabble.com/Function-to-truncate-a-file-td7684482.html
>>
>> If you need a portable solution, you can use Lua/APR
>> (http://peterodding.com/code/lua/apr/docs/#file:truncate) or fs
>> library (https://luapower.com/fs; requires FFI).
>>
>> Paul.
>>
>> On Thu, Jun 20, 2019 at 4:21 PM Sergey Kovalev <[hidden email]> wrote:
>> >
>> > How to trim existing file from 2Gb to 1Gb using lua io?
>> >
>>

Reply | Threaded
Open this post in threaded view
|

Re: Ho to trim file in lua?

Philippe Verdy
This works only for increasing the filesize; the question was about *trimming* it to reduce its size.
And anyway there's no specification when increasing the size with "file:seek" that the file will not be completely written (with huge I/O load and excessive flushing of I/O caches) instead of being just be preallocated (with clusters implicitly marked as full of null bytes).
So your solution is not that good. Filesystems can do things much better (possibly via an "ioctl" API).

Le lun. 24 juin 2019 à 20:43, Sergey Kovalev <[hidden email]> a écrit :
-- preallocate 1Gb
f=io.open("dummy.bin","wb") or error()
f:seek("set",1024*1024*1024-1)
f:write("1")
f:close()

пт, 21 июн. 2019 г. в 21:14, Philippe Verdy <[hidden email]>:
>
> May be some native extension can add this functionality to existing file objects (by adding i/o methods, or a way to pass native I/O requests via some escape mechanism that the extension will check itself.
>
> There are other features missing infile I/O, notably the possibility of prealocating empty space (without really having to write it on disk: it can jsut be marked in the filesystem allocation table has being allocated, but also implictly full of zeroes, freeing the OS from having to make physical reads as well, and it is very useful as well for "virtual" files. Another possibility is to virtualize a file's version (with separate commits of sectors/clusters into versioned branches), or the possibility of virtualizing multiple distinct files to share the same physical space by default, also virtualized when one file is written but not the other).
>
> Various filesystems have already implemented these, notably to better support virtual machines with lower cost and improved performance by reduced physical I/O; it is already the case on Linux (with recent versions of Ext4, including for Android devices) and Windows (with NTFS), and most probably as well on MacOS. And any OS that has now support for efficient implementation of virtual machines have added such support in at least one of their supported filesystems, or have added one which can also be used for other applications (including for installing the OS itself and maintaining it securely with "instant snashots").
>
> I wonder why POSIX did not try to extend the minimum set of file I/O API that a decent OS should support (or that can probably be easily emulated by small plugins in the implementation of virtual machines and scripting engines like Lua, even if sometimes it may be much slower than when using the native filesystem support).
>
>
> Le ven. 21 juin 2019 à 01:37, Paul K <[hidden email]> a écrit :
>>
>> > How to trim existing file from 2Gb to 1Gb using lua io?
>>
>> Lua I/O doesn't provide any function to truncate a file. I supported
>> suggestion to add one. See this thread for a previous discussion on
>> this topic: http://lua.2524044.n2.nabble.com/Function-to-truncate-a-file-td7684482.html
>>
>> If you need a portable solution, you can use Lua/APR
>> (http://peterodding.com/code/lua/apr/docs/#file:truncate) or fs
>> library (https://luapower.com/fs; requires FFI).
>>
>> Paul.
>>
>> On Thu, Jun 20, 2019 at 4:21 PM Sergey Kovalev <[hidden email]> wrote:
>> >
>> > How to trim existing file from 2Gb to 1Gb using lua io?
>> >
>>
Reply | Threaded
Open this post in threaded view
|

Re: Ho to trim file in lua?

Philippe Verdy
Using "wb" also has the bad effect of unallocating the file completely and reallocating it from zero. If that file still had pending sectors to write, the reallocation will be elsewhere on disk instead of reusing the exising storage space when possible (this is not possible if the file is "versioned" by the filesystem, but if it is, each user or other versioning context would then cause the whole file to be rewritten instead of just parts that are actually written by each user).
And it's also the filesystem that controls if versioning must occur (possibly with transactional support) and that determines how to split the fragments to version (cluster by cluster? sector by sector? smaller units? minimum groups of clusters?)

Le lun. 24 juin 2019 à 21:27, Philippe Verdy <[hidden email]> a écrit :
This works only for increasing the filesize; the question was about *trimming* it to reduce its size.
And anyway there's no specification when increasing the size with "file:seek" that the file will not be completely written (with huge I/O load and excessive flushing of I/O caches) instead of being just be preallocated (with clusters implicitly marked as full of null bytes).
So your solution is not that good. Filesystems can do things much better (possibly via an "ioctl" API).

Le lun. 24 juin 2019 à 20:43, Sergey Kovalev <[hidden email]> a écrit :
-- preallocate 1Gb
f=io.open("dummy.bin","wb") or error()
f:seek("set",1024*1024*1024-1)
f:write("1")
f:close()

пт, 21 июн. 2019 г. в 21:14, Philippe Verdy <[hidden email]>:
>
> May be some native extension can add this functionality to existing file objects (by adding i/o methods, or a way to pass native I/O requests via some escape mechanism that the extension will check itself.
>
> There are other features missing infile I/O, notably the possibility of prealocating empty space (without really having to write it on disk: it can jsut be marked in the filesystem allocation table has being allocated, but also implictly full of zeroes, freeing the OS from having to make physical reads as well, and it is very useful as well for "virtual" files. Another possibility is to virtualize a file's version (with separate commits of sectors/clusters into versioned branches), or the possibility of virtualizing multiple distinct files to share the same physical space by default, also virtualized when one file is written but not the other).
>
> Various filesystems have already implemented these, notably to better support virtual machines with lower cost and improved performance by reduced physical I/O; it is already the case on Linux (with recent versions of Ext4, including for Android devices) and Windows (with NTFS), and most probably as well on MacOS. And any OS that has now support for efficient implementation of virtual machines have added such support in at least one of their supported filesystems, or have added one which can also be used for other applications (including for installing the OS itself and maintaining it securely with "instant snashots").
>
> I wonder why POSIX did not try to extend the minimum set of file I/O API that a decent OS should support (or that can probably be easily emulated by small plugins in the implementation of virtual machines and scripting engines like Lua, even if sometimes it may be much slower than when using the native filesystem support).
>
>
> Le ven. 21 juin 2019 à 01:37, Paul K <[hidden email]> a écrit :
>>
>> > How to trim existing file from 2Gb to 1Gb using lua io?
>>
>> Lua I/O doesn't provide any function to truncate a file. I supported
>> suggestion to add one. See this thread for a previous discussion on
>> this topic: http://lua.2524044.n2.nabble.com/Function-to-truncate-a-file-td7684482.html
>>
>> If you need a portable solution, you can use Lua/APR
>> (http://peterodding.com/code/lua/apr/docs/#file:truncate) or fs
>> library (https://luapower.com/fs; requires FFI).
>>
>> Paul.
>>
>> On Thu, Jun 20, 2019 at 4:21 PM Sergey Kovalev <[hidden email]> wrote:
>> >
>> > How to trim existing file from 2Gb to 1Gb using lua io?
>> >
>>
Reply | Threaded
Open this post in threaded view
|

Re: Ho to trim file in lua?

Philippe Verdy
As well we can imagine a transactional filesystem will provide transaction APIs, 
 file:beginx()
 if file:commitstartx() then ... end
 file:commitx()
 file:rollbackx()
The idea being that any file:write() will be part of a transaction started by file:beginx() and automatically rollbacked if it is closed before using file:commitx() (or the program is aborted, or if the system crashes and is rebooted to remount the filesytem)

All these file:write()s will be made to a journaled version allowing safe recovery.

The additional file:commitstartx() API would be necessary to allow concurrent transactions in any order with two phase commits, it tests if the transaction can be commited without creating a deadlock condition, allowing the program to either rollback and possibly rollback other transactions blocked by other owned file-locks.

In such situation, the journal may use subcluster allocation in a serial stream owned by the current context (which still uses it on file:read() to get consistant reads). In that case there will be minimum I/O overhead. when writing to the journal (and once you commit the transaction, the space allocated in the journal stream can be reused by the same context/user for other transactions, as long as the file remains open for that context.

Also note that your solution of writing one static byte at the final position is bad: it overwrites that byte whose initial value is then lost, so it is not a "truncation" of the file. Finally you chosen byte ("1") would force the journaling filesystem to allocate space for it even if that space was first "virtual". If you choose a byte value it should be a NULL byte (so that no space is allocated in the virtual file storage), even if it also means that any prior non-NULL byte will be also lost (virtually) as if it was overwritten (if that byte was in a non-virtual sector/cluster was not full of NULLs, you'll get an allocation for that sector/cluster when it is committed to disk on a transactional filesytem, or that NUL byte will still be physically written over the previous byte if the filesystem is not versioned, even if that file system is not transactional).

For all these reasons, file APIs should be extended for supporting:
- journaling
- transaction (including two-phase commits)
- versioning
- virtual preallocation and truncation at any seek position (even a very large one)
- possibly creating embedded subtransactions (only if there's support for two-phase-commits).
And the basic file:open(), fileread(), file:write(), file:seek(), and file:close() operations from POSIX are not sufficient for modern apps. All these can be emulated easily on filesystems that are neither transactional, nor journaling, nor virtualized.



Le lun. 24 juin 2019 à 21:32, Philippe Verdy <[hidden email]> a écrit :
Using "wb" also has the bad effect of unallocating the file completely and reallocating it from zero. If that file still had pending sectors to write, the reallocation will be elsewhere on disk instead of reusing the exising storage space when possible (this is not possible if the file is "versioned" by the filesystem, but if it is, each user or other versioning context would then cause the whole file to be rewritten instead of just parts that are actually written by each user).
And it's also the filesystem that controls if versioning must occur (possibly with transactional support) and that determines how to split the fragments to version (cluster by cluster? sector by sector? smaller units? minimum groups of clusters?)

Le lun. 24 juin 2019 à 21:27, Philippe Verdy <[hidden email]> a écrit :
This works only for increasing the filesize; the question was about *trimming* it to reduce its size.
And anyway there's no specification when increasing the size with "file:seek" that the file will not be completely written (with huge I/O load and excessive flushing of I/O caches) instead of being just be preallocated (with clusters implicitly marked as full of null bytes).
So your solution is not that good. Filesystems can do things much better (possibly via an "ioctl" API).

Le lun. 24 juin 2019 à 20:43, Sergey Kovalev <[hidden email]> a écrit :
-- preallocate 1Gb
f=io.open("dummy.bin","wb") or error()
f:seek("set",1024*1024*1024-1)
f:write("1")
f:close()

пт, 21 июн. 2019 г. в 21:14, Philippe Verdy <[hidden email]>:
>
> May be some native extension can add this functionality to existing file objects (by adding i/o methods, or a way to pass native I/O requests via some escape mechanism that the extension will check itself.
>
> There are other features missing infile I/O, notably the possibility of prealocating empty space (without really having to write it on disk: it can jsut be marked in the filesystem allocation table has being allocated, but also implictly full of zeroes, freeing the OS from having to make physical reads as well, and it is very useful as well for "virtual" files. Another possibility is to virtualize a file's version (with separate commits of sectors/clusters into versioned branches), or the possibility of virtualizing multiple distinct files to share the same physical space by default, also virtualized when one file is written but not the other).
>
> Various filesystems have already implemented these, notably to better support virtual machines with lower cost and improved performance by reduced physical I/O; it is already the case on Linux (with recent versions of Ext4, including for Android devices) and Windows (with NTFS), and most probably as well on MacOS. And any OS that has now support for efficient implementation of virtual machines have added such support in at least one of their supported filesystems, or have added one which can also be used for other applications (including for installing the OS itself and maintaining it securely with "instant snashots").
>
> I wonder why POSIX did not try to extend the minimum set of file I/O API that a decent OS should support (or that can probably be easily emulated by small plugins in the implementation of virtual machines and scripting engines like Lua, even if sometimes it may be much slower than when using the native filesystem support).
>
>
> Le ven. 21 juin 2019 à 01:37, Paul K <[hidden email]> a écrit :
>>
>> > How to trim existing file from 2Gb to 1Gb using lua io?
>>
>> Lua I/O doesn't provide any function to truncate a file. I supported
>> suggestion to add one. See this thread for a previous discussion on
>> this topic: http://lua.2524044.n2.nabble.com/Function-to-truncate-a-file-td7684482.html
>>
>> If you need a portable solution, you can use Lua/APR
>> (http://peterodding.com/code/lua/apr/docs/#file:truncate) or fs
>> library (https://luapower.com/fs; requires FFI).
>>
>> Paul.
>>
>> On Thu, Jun 20, 2019 at 4:21 PM Sergey Kovalev <[hidden email]> wrote:
>> >
>> > How to trim existing file from 2Gb to 1Gb using lua io?
>> >
>>