[Mod] Exchange stuff between servers [Interkom]

[Gundul]

! CAUTION !

This mod is wip and in development.
Use at your own risk.
No garantees, no support, I am not responsible for any loss. :D
But do not worry, I have this running for more than half a year and
all my servers and harddiscs are still alive

Dependencies: none - (adds button to unified_inventory if available)
License:

+ Spoiler

DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
Version 2, December 2004

Copyright (C) 2004 Sam Hocevar
14 rue de Plaisance, 75014 Paris, France
Everyone is permitted to copy and distribute verbatim or modified
copies of this license document, and changing it is allowed as long
as the name is changed.

DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

0. You just DO WHAT THE FUCK YOU WANT TO.

This mod works, proofed on the following connected servers:
"Jungle"
"Lilly in the valley"
"LouisZ" (not in serverlist)

You need to setup a synchronized folder if you want to connect servers.
By default this folder must be in your world directory named "Lilly"
for example ~/.minetest/worlds/myworld/Lilly

With this mod it is possible to transfer stuff from one server to an other. It also works
on one server only, but /give command is a bit faster :)
All can be send, except tools or nodes containing metadata. You are free to use what ever
mods you want on each server because interkom will know what is known and which nodes
not. Unknown things will return to sender.


Sorry no pictures as this mod only works with chatcommands so far:

/interkom
This will show you a chart of all connected servers with its players

/pm player,server,message
Will send a private message to player@server

/stuff player,server,stuff
Will send "stuff" to player@server, where stuff is an itemstack like "default:stone 25"

/ok
Will accept what a player wants to send you and puts him on the whitelist

/bl player@server
puts player@server on the blacklist again

So far it works on above servers, I will add other features in the future and maybe a
gui too. Any suggestions post here please.

download here: https://github.com/berengma/interkom/archive/master.zip
Sourcecode here: https://github.com/berengma/interkom


Gundul

[sofar]

This code looks super iffy. A fun read to find input validation issues so people can send arbitrary stuff they don't have to players on other servers. Are you sure you're checking all the input correctly? Did you protect against races and properly lock the transfer files?

You should post the dependencies and license in the forum thread. See sticky topic.

[Gundul]

This code looks super iffy.

just for the record:

-I do not post something "iffy" in mods section, this works for months without iussues on my two public servers
-We are here in wip mods section.Wip means: work in progress.
Do not expect something here what would not needed to be worked on.

A fun read to find input validation issues so people can send arbitrary stuff they don't have to players on other servers. Are you sure you're checking all the input correctly? Did you protect against races and properly lock the transfer files?

Exaggerated a bit ?
But I am always happy other people have fun with my developments. Any proove, example maybe ?
I am repeating myself, this mod is running for months on my servers without trouble.

[sofar]

This code looks super iffy.

just for the record:

-I do not post something "iffy" in mods section, this works for months without iussues on my two public servers
-We are here in wip mods section.Wip means: work in progress.
Do not expect something here what would not needed to be worked on.

A fun read to find input validation issues so people can send arbitrary stuff they don't have to players on other servers. Are you sure you're checking all the input correctly? Did you protect against races and properly lock the transfer files?

Exaggerated a bit ?
But I am always happy other people have fun with my developments. Any proove, example maybe ?
I am repeating myself, this mod is running for months on my servers without trouble.

If I reply with concerns then it means (1) I generally like the code/mod/idea, and (2) I care enough to voice my concerns in case that I think it's warranted. It's not a personal attack, obviously, I never said "Oh gundul has the nerve to tickle a wombat!", instead I said "this code looks suspect (iffy)" with the intent of making you critically think about whether this is actually secure or not.

It obviously works, I don't doubt it. What I doubt is that if players send requests at the same time that both actually arrive intact. Since the receiving server can't verify that the items are in the senders' inventory, it's not hard to think of a possible exploit to generate some message that gets the receiver lots of items that the sender never had, or just corrupt the messages. There's also some splitting of user input going on without escaping, so there's a possibility of people throwing weird characters at you that you didn't think of ("\n") and may end up in your transfer file.

It's not my code, if you don't want to secure it more, that's your perogative. But now that you've posted the code, if there is an exploit possible, anyone can find it.

[Krock]

I wondered how "iffy" the code actually is and only figured out some minor issues:

1) Unreliable file writing. I'd rather use `minetest.safe_write_file` to ensure everything is written correctly in case the server is not shut down correctly.
2) Writing to the file at the same time: If server 1 writes to the file whereas server 2 deletes an entry from the file, there may be a file corruption. Not sure though, how this could be solved.
3) If you ever allow metadata: New lines and commas would break the system. There's no escape handling in your custom string.split implementation and new lines would destroy the format. Luckily you treat items with metadata like tools (disallowed), even though they could be stackable.

It was however interesting to see this mix of tabs and spaces. I think you're using four spaces, but the editor does not convert existing tabs when you indent the code.

[Gundul]

If I reply with concerns then it means (1) I generally like the code/mod/idea, and (2) I care enough to voice my concerns in case that I think it's warranted. It's not a personal attack, obviously, I never said "Oh gundul has the nerve to tickle a wombat!", instead I said "this code looks suspect (iffy)" with the intent of making you critically think about whether this is actually secure or not.

Sorry for getting you wrong, engish is not my native language and "iffy" is not the word we would use in a situation like this.

It obviously works, I don't doubt it. What I doubt is that if players send requests at the same time that both actually arrive intact. Since the receiving server can't verify that the items are in the senders' inventory, it's not hard to think of a possible exploit to generate some message that gets the receiver lots of items that the sender never had, or just corrupt the messages. There's also some splitting of user input going on without escaping, so there's a possibility of people throwing weird characters at you that you didn't think of ("\n") and may end up in your transfer file.

The security is always at my mind. But it is only about sharing some stuff between servers. Worst that could happen is someone getting lots of stuff he did not have. And I get the debug for free and can work with that. :)
I already updated some parts of the code checking more detailed for nodenames.

It's not my code, if you don't want to secure it more, that's your perogative. But now that you've posted the code, if there is an exploit possible, anyone can find it.

I will make it as secure as I am able to do, that is why I post it now and not already did when I first put the mod
online. That is also why I put the warning caution and do not describe more in detail how to synchronize folders.
Everybody must decide this for his own. Not to forget that exploits also can have positve effects when server and
running software are well monitored.

[Gundul]

I wondered how "iffy" the code actually is and only figured out some minor issues:

1) Unreliable file writing. I'd rather use `minetest.safe_write_file` to ensure everything is written correctly in case the server is not shut down correctly.

Thank you, I did not know that exists in the minetest api.

2) Writing to the file at the same time: If server 1 writes to the file whereas server 2 deletes an entry from the file, there may be a file corruption. Not sure though, how this could be solved.

I had this in my mind. But I must confess I have no solution for this, too. I was thinking that the software which synchs the folders would take care of this maybe.

3) If you ever allow metadata: New lines and commas would break the system. There's no escape handling in your custom string.split implementation and new lines would destroy the format. Luckily you treat items with metadata like tools (disallowed), even though they could be stackable.

I do not plan to allow metadata. This is not wanted now and in the near future.
Maybe all this can be done with a gui in the future, but only if this really runs reliable.

Items with metadata could be stackable, yes. But in that case the metadata would be lost if sent to an other server.

It was however interesting to see this mix of tabs and spaces. I think you're using four spaces, but the editor does not convert existing tabs when you indent the code.

Sorry, which tabs and spaces you mean ?

[Krock]

Items with metadata could be stackable, yes. But in that case the metadata would be lost if sent to an other server.

Ah okay. I didn't look at the code for a too long time to notice that behaviour.

Sorry, which tabs and spaces you mean ?

These particular lines are interesting: https://github.com/berengma/interkom/bl ... #L233-L239
space-only indents: 3/7 lines
1x tab, spaces: 1/7 lines
tab-only indents: 3/7 lines

Not important in terms of functionality but looks a bit as if you copies&pasted multiple mods into this one.

[sofar]

My initial thoughts on this mod are to secure the wire protocol and not use files as they are obviously prone to locking issues (as lua does none) and 2 servers could write the same file at the same time. A better implementation would be to use sockets to do all the communication, where each server has a "receive" socket, and the sender transmits by sending some json encoded structural content.

The chatcommands and parsing of input are difficult things to get right (does it handle utf8 properly?) so I would think about replacing the chatcommands with a properly secured formspec where players need to put the actual items in a send slot, which also takes the trouble of knowing the item names away for players. You could use the sockets to send player lists to each server so that players know who is online or not and have the form display this, or maintain a friend list or something like that.

[Lone_Wolf]

2) Writing to the file at the same time: If server 1 writes to the file whereas server 2 deletes an entry from the file, there may be a file corruption. Not sure though, how this could be solved.

I had this in my mind. But I must confess I have no solution for this, too. I was thinking that the software which synchs the folders would take care of this maybe.

I'm not sure of how everything works in your system but you might be able to create a 'status file' for each server in the folder and set it up so that before a server writes anything it checks the other server's status files.
If none of the other servers is writing have it set the contents of its status file to something that says it's writing to a file.
Make it write to the file it was originally going to write to, and when it finishes you can make it rewrite it's status file to something saying it's not writing

I hope that's readable...

[sofar]

I'm not sure of how everything works in your system but you might be able to create a 'status file' for each server in the folder and set it up so that before a server writes anything it checks the other server's status files.
If none of the other servers is writing have it set the contents of its status file to something that says it's writing to a file.
Make it write to the file it was originally going to write to, and when it finishes you can make it rewrite it's status file to something saying it's not writing

I hope that's readable...

Now you just created a second locking problem with the state file. :)

[Gundul]

Make it write to the file it was originally going to write to, and when it finishes you can make it rewrite it's status file to something saying it's not writing

I hope that's readable...

Thank you it was readable :)
But I think sofar already answered this. :)

[Gundul]

My initial thoughts on this mod are to secure the wire protocol and not use files as they are obviously prone to locking issues (as lua does none) and 2 servers could write the same file at the same time. A better implementation would be to use sockets to do all the communication, where each server has a "receive" socket, and the sender transmits by sending some json encoded structural content.

The chatcommands and parsing of input are difficult things to get right (does it handle utf8 properly?) so I would think about replacing the chatcommands with a properly secured formspec where players need to put the actual items in a send slot, which also takes the trouble of knowing the item names away for players. You could use the sockets to send player lists to each server so that players know who is online or not and have the form display this, or maintain a friend list or something like that.

You are right with the formspecs. I chose the chatcommands because experience from my servers says
players are lazy in writing. That way I could test my mod online on my servers more relaxed in the beginning.

I must confess I have very little knowledge about sockets, but that can change. Always happy to learn something.
A quick search returned a "lua socket api" for lua 5.1. No idea if that could help. If someone knows any good source
or book please feel free to post/tell me.

The next steps for interkom will be:

- use minetest.safe_write_file if possible
- restrict use of chatcommands to players with interkom(server ?) priv
- add formspecs to handle items and send them to connected servers
- find a way to use sockets

These steps do not have to be static. I am happy for any serious idea someone could have.

[Gundul]

created a new branch using minetest.safe_file_write
https://github.com/berengma/interkom/tree/savewrite

[Gundul]

Finished the gui. No more writing needed. Just type /stuff. On the left see the connected servers and on the right
you can see the players on that sever. To the four slots in the middle you can move the stuff you want to send.
Press send and watch chat messages if everything went well.


Now we have pictures :)


I will add support for unified_inventory shortly, so you can use directly from your inv.

[Otter]

I just want to support that this is quite tested and seems to work without hitch. I am quite particular about things disappearing in mods. The exchange using this software ends up being quite robust and when there were issues, the items did not get sent.
I have sent quite a bit of stuff between the servers. The stacks can be large on these servers so a lot can be sent at once. Between the servers there is a strong advantage to exchanges. Some things common enough on one are quite rare on the other. This worked out well.

It was an entirely different mod and not one made by Gundul that showed a bug and deleted weeks of productivity. In retrospect, sending your most valuable resources from multiple servers to a single server is probably foolish. Loosing large stacks of diamond, and mese blocks that you spent weeks accumulating can be pretty discouraging. Especially when you need them to continue researching how to use the mod that destroyed the stuff. :)

[Gundul]

I just want to support that this is quite tested and seems to work without hitch. I am quite particular about things disappearing in mods. The exchange using this software ends up being quite robust and when there were issues, the items did not get sent.
I have sent quite a bit of stuff between the servers. The stacks can be large on these servers so a lot can be sent at once. Between the servers there is a strong advantage to exchanges. Some things common enough on one are quite rare on the other. This worked out well.

I can confirm, no issues so far since half a year running this on my servers. No lost stuff at all.
Now even everything can be sent, not only default:xxx and
with the new gui pretty easy for everyone to handle.

It was an entirely different mod and not one made by Gundul that showed a bug and deleted weeks of productivity. In retrospect, sending your most valuable resources from multiple servers to a single server is probably foolish. Loosing large stacks of diamond, and mese blocks that you spent weeks accumulating can be pretty discouraging. Especially when you need them to continue researching how to use the mod that destroyed the stuff. :)

Maybe interesting to know which mod that was ? I did not know that there was a mod like mine before.

[Gundul]

Fixed bug today when sending with gui to new players.
You will be asked to whitelist the sender and if you /ok
all items will be added to your inv. No more losses !


Gundul

[Gundul]

added button to unified_inventory for easy access :