[Mod]FFIopt [ffiopt]

[EvidenceBKidscode]

FFIopt optimizes access to large amounts of data via the Lua API and reduces Lua memory footprint.

License: LGPL v3

Dependancies: None of mods but needs engine to be patched

Source: https://github.com/EvidenceBKidscode/ffiopt

Description

Lua API methods that transmit large amount of data (VoxelManip:get_data, PerlinNoiseMap:get3dMap_flat) copies data from C++ structures to Lua tables. This copy is quite slow and consumes a lot of Lua memory.

FFIopt replaces this copy by a direct access from Lua to C++ data using FFI library. The engine has to be patched to expose some C++ pointers to Lua API.

If engine correctly patched, FFIopt hacks VoxelManip and PerlinNoiseMap to use FFI rather than Lua tables.

Result is :

• Faster data retreiving and storing ;
• Slower table access (essencially due to use of metatable avoiding mods code change) ;
• Lower Lua memory footprint (avoiding many OOM errors) ;

How to use it

• Use LuaJIT.
• Patch the engine to add VoxelManip:get_data_ptr, PerlinNoiseMap:get_result_size and PerlinNoiseMap:get_result_ptr methods to the API.
• Install and enable ffiopt mod.
• On starting Minetest, ffiopt will tell if it is able to optimise voxel manips and perlin noise.
• Nothing to change in mods code.

Experiments

FFIopt can be tested through ffiopt_demo mod. Here is what we obtained.

Voxel Manip

Without FFIopt, in the demo, 30 % of the time is spend in get_data and another 30 % is spent in set_data. Lua processing takes 20 % and read/write the remaining 20 %.

So most of the time is spent in converting C++ data to Lua table.

With FFIopt, Lua takes 68 % of the time and read/write 32 % (get_data and set_data don’t do anything).
Lua processing time is increased by 75 %, but total time is reduced by 50 %.

Furthermore, without FFIopt, launching demo 9 times leaded to 3 OOM crash. With FFIopt, we have been able to launch more than 20 times the demo with no crash.

Perlin noise

With FFIopt, Lua processing increased by 50 % but total time reduced by 15 %.

Lua time is increased mostly because of the use of a metatable. This can be avoided but the mod code has to be changed (C++ indexes starts from 0, not 1) and secured to avoid out of range indexes.

[runs]

Why is not this in the official engine yet?

Are you a https://kidscode.com developer?

[Pyrollo]

Are you a https://kidscode.com developer?

Obviously.

Have a look here: viewtopic.php?f=3&t=22090

[Festus1965]

I am using it actual, got the patch done, mod active
thanks, I was reading that about the FFI, but no good enough to make something ...

[runs]

I am using it actual, got the patch done, mod active
thanks, I was reading that about the FFI, but no good enough to make something ...

You said that consumes much more memory...

[Festus1965]

You said that consumes much more memory...

Nope,
I wrote that compare to just before used configuration for 6 days, the minimum was 66 MB,
then the one just for a view hours used 51 MB at start (no [Mod] time, updated [Mod] EUban),
and this new one used once (mean not confirmed, as also [Mod] time is back) was using more, as 78 MB
but I updated the collect-garbage result after one hour and there are 52 MB used now.

So as there are only 2 changes left: update [mod] EUban and new [Mod] ffiopt - what is the reason ?
I think that might be more about better EUban, but how often I have to start both versions (with/-out ffiopt) to be sure ?

[Festus1965]

but I found another problem
description here: server ffiopt and worldedit seam have a problem with other

WorldEdit didn't work when ffiopt true

[Pyrollo]

Mmmhh..

Worldedit uses voxelmanip in a non usual way and does not use VoxelManip:get_data. Some change required to make it compatible.

[Pyrollo]

New issue opened: https://github.com/EvidenceBKidscode/ffiopt/issues/1

If you have any issue, please don't hesitate to report them in Github.

[pgimeno]

Sorry, I'm no longer registered in Microsoft GitHub and I'm not planing to return, so I can't submit issues there.

I see several issues with this mod.

First and foremost, security. I see two issues in this respect:

- It's exposing the FFI interface publicly, which basically defeats the purpose of having mod security. Any mod can, through the exposed FFI interface, call any C library function including, but not limited to, fopen and fwrite.
- Besides that, the pointer to the data is available in the table itself. This makes it possible (through use of large or negative indexes) to access any arbitrary memory position, which could potentially expose passwords or other stuff to untrusted mods.

Next is the use of a long int, and subsequently a Lua number (double precision float), to hold a pointer. This is not portable, and it may cause issues in some platform. It may turn out, by pure luck, to work fine in all platforms that Minetest supports, though.

Then, the benchmarks do not use the buffer version of the functions, which are designed to prevent object creation and internal reallocation, therefore it's not really a very fair comparison for well-written mods.

Lastly, it does not patch the functions with the new spellings: get_2d_map_flat et al.

I wrote a PR for Minetest that exposed C functions that could be called through FFI, to let modders handle this manually, but it was not taken with a lot of enthusiasm by the core developers. See https://github.com/minetest/minetest/pull/6863 (all appearances of 'ghost' I can see as of this date are my posts).

[rubenwardy]

I wrote a PR for Minetest that exposed C functions that could be called through FFI, to let modders handle this manually, but it was not taken with a lot of enthusiasm by the core developers. See https://github.com/minetest/minetest/pull/6863 (all appearances of 'ghost' I can see as of this date are my posts).

I really like this feature and it had the potential reduce OOMs and improve performance - but you yourself said that the performance improvement was negligible

[pgimeno]

I really like this feature and it had the potential reduce OOMs and improve performance - but you yourself said that the performance improvement was negligible

Not negligible, but small in comparison with the processing time required in most use cases (Lua mapgens that process noise for landscaping).

That was actually paramat's point: https://github.com/minetest/minetest/pu ... -354859130 and I agreed (edit: https://github.com/minetest/minetest/pu ... -355646707), but even then I see value in this, because of moving the allocation outside of Lua memory.

[Pyrollo]

Hi pgimeno, thanks a lot for your feedback, many interesting things :)

We'll study that closely, trying make mod evolve.

Then, the benchmarks do not use the buffer version of the functions, which are designed to prevent object creation and internal reallocation, therefore it's not really a very fair comparison for well-written mods.

I did not take care of this aspect in the benchmark because in various studied options, it turned out that every time passing values through Lua stack was used to retrieve map data, things were slow (tryed to implemet something like get(index) on the C++ side but the get argument value/returning map value took almost the same time than populating the table).

For our purpose (changing many nodes at once) using FFI had a noticeable change on performances (but at first we disabled the light calculation on save, way to slow and not necessary for us).

On a memory point of view I totally agree with you. Data should not be duplicated to Lua memory.