Technical discussion about cheating and hacked clients

[Ferret2]

Thread for for general discussion about all the ways in which minetest makes it technically feasible or even easy to cheat for an online player, including hacked client programs, hacked client textures, etc. Discussion includes identification of various technical problems, and also includes all kinds of ideas for improvement.

About me: After having previously played minetest in singleplayer mode for a few years, I started playing online on a server only a few weeks ago. As a very naive new online player, I was surprised to discover that minetest seems to have a significant problem with hacked clients.

Definition of the problem: It seems that technically, minetest makes it rather easy for clients to cheat. Examples are: A player with a hacked client can give himself lots of privileges including flying and moving trough walls, and also a player with a hacked client apparently can make himself unkillable (hitpoints do not seem to go down when attacked). In addition, just now I also saw on https://www.youtube.com/watch?v=wCK3I4HbRKw that simply by changing the texture for default stone on the client side into a transparent image, you can enable yourself to look through stone (so that you can see distant caves and the mobs and ores in them). These easily achieved hacks in a player's client installation give the cheater superpowers which make it totally impossible for normal players (who have not hacked their clients) to compete against him. Because these cheats are technically easy to do, cheating players with hacked clients are apparently a frequent occurrence on servers. This spoils the online game for everyone who does not like to cheat.

This thread is a spin-off from the JT2 server thread https://forum.minetest.net/viewtopic.php?t=15334&start=1400 from the discussion about hacked clients emerging there from about Mar 15, 2019.

I would like to also point to the following two excellent posts on the forum https://github.com/minetest/minetest/issues/6114

* The post by raymoo from Jul 10, 2017
* The post by beyondlimits from Aug 7, 2017

which already state what in my view seems to be the only viable direction of solution, namely the software should be changed so that the server checks everything (that the players move legally, etc).

[Ferret2]

Hello 843jdc,

Many thanks for your post in the JT2 thread. To help keep that thread on topic, I'll reply in the present thread.

Welcome to the anarchy world of Minetest.

Heh:-). That does seem to be an accurate description. I bumped into this hacker problem totally naively. I hadn't expected the hacker problem when I started playing online about 5 weeks ago. The hacker problem is currently sort of totally changing my perspective on minetest.

Thank you for your ideas. It's great to discuss this.

First I have to ask some questions, namely for understanding exactly what you mean with the following:

I agree the server should track player actions much more closely than it does. Servers send out a massive amount of data even for only a few players. But receive a "blip" of data from clients. So tracking players closely will introduce a lot of "player position resets" once the server command finally gets a slot in the transmit queue. Players (ME!) get ticked off fast when that happens often.

I agree of course that the additional tasks that we will need the server to perform to limit the scope for hackers, that these additional tasks should not be so heavy that the game becomes too slow for players to enjoy.

First question: What do you mean with "the server command"? Is that the request that we are envisaging the client wil send to the server, for requesting the player's avatar to move from one node into a neighbouring node?

Second: What does the term "player position reset" mean?

Third: I do not understand how exactly that communication bottleneck works that you are addressing. Do you mean to say that if there is a lot of traffic from server to client, that this will block the traffic from that client to the server?

(Note: I am afraid I am not yet familiar with the minetest source code. I have studied the map database format though, and I understand that the server simply sends complete map blocks to the client. And also I have a little (but not a huge) experience in programming TCP/IP servers and clients. I realize I will probably have to dig up my TCP/IP book presently.)

[Ferret2]

A fix for viewing into someone else's locked chests should be relatively easy: if the client isn't used by the chest's owner don't send metadata of the contents.

Indeed. The general idea should be, I think, that the server should send to a client only that part of the data that the client is allowed to see. This means that the server should not send to the client exactly the data as it is represented (stored) on the server, but instead it should only send "censored" data, i.e. data from which everything is removed that that particular client is not allowed to see.

I.e. I think that logically speaking there should be a "filter" component in the data stream from the server to the client, placed between the data as represented (stored) on the server and the data that is sent to the client. On the server, there is an outgoing data stream for every connected client. Each of these outgoing data streams has its own Filter instance.

Data flow diagram:

Code: Select all

        +------------+                 ,-----------.        +--------------+
        | Data       |  ------*---->   | Filter 1  |  ----> | Data sent    |
        | on server  |        |        `-----------'        | to client 1  |
        +------------+        |                             +--------------+
                              |
                              |        ,-----------.        +--------------+
                              *---->   | Filter 2  |  ----> | Data sent    |
                              |        `-----------'        | to client 2  |
                              |                             +--------------+
                              |
                              ...


Each Filter instance knows the ID (username) and the position (coordinates) of the player it is connected to, and from that information the Filter knows how to perform the operation of censoring the data for that client. The central data as it is stored on the server (leftmost box in the diagram) should be global and not specific to any one player (i.e. exactly as it is at the moment in the current minetest implementation).

The operation performed by the Filters should obviously not be too time consuming. Simply copying the contents of a map block from the central database to a buffer, and during that copying operation leaving out the data that the client is not allowed to see, should (I think) be fast, as long as the decisionmaking about which node the client is allowed to see (and about which past of the data associated with the node the client is allowed to see) is simple. Checking for chest ownership doesn't sound complicated.

[Ferret2]

If a node is dug, server then sends out correct node type of revealed nodes. Server and network lag big factor here.

Yes, if a player digs a node, I agree that then somehow the data about the revealed nodes has to be sent to the client.

In the original scheme I had in mind (see my previous posts), this would mean that whenever a player digs a node, the whole map block has to be re-sent from server to client. In the old map block (that the client had received previously to the dig), the newly revealed nodes were censored out. In the new map block (that the server has to now send to the client), these newly revealed nodes are left in and are thus now visible to the client.

Would that be too slow, to send a whole map block whenever a player digs a node? I'm sure this can be optimized. Most of the time, the dig wouldn't hit a cave (i.e. air on the other side). So in that case if I count correctly the number of new revealed nodes can not be more than 5 (1 forward in digging direction, 4 to the sides), so in that case the server would logically speaking only have to send at most 5 new nodes, which is not much data. Of course, if the server is slow to respond, then the player will experience lag there. Therefore the conclusion is that in a hacker-safe version of minetest it is inevitable that (with a slow server) there will be server lag whenever you dig a node. (However, as long as you only move around without digging, there will be no additional server lag compared with current minetest.) What do you think, will that make the game unplayable/impractical?

If the dig hits a cave, then much more data has to be sent, because in general it can be a big cave, i.e. revealing a lot of new stuff to the player. But hitting a cave doesn't happen often, so I think that lag is acceptable in that case.

Maybe generate ores as the player digs. Though other players might not see those ores. Player-specific???

I'm pretty sure this is not an issue (correct me if I am wrong). The central data on the server (the map database on the server) simply generates a new map block whenever needed, i.e. whenever a map block needs to be accessed that is not yet in the database. If I understand correctly, this is exactly how minetest operates at the moment. The problem is not map generation: the central map in the map database on the server contains all data (all ores) and is not specific to any one player. Each player simply gets sent map blocks in which the nodes that he is not allowed to see are censored out. I.e. the central map doesn't change as a player digs; the only thing that changes as a player digs is the *view* which that player has on the central map.

More replies on your post from the JT2 thread maybe later.

[micheal65536]

In b4 this thread gets deleted under the "no discussion of cheating" rule.

Because seriously, Minetest needs an open discussion about the technical aspects of hacking/cheating and how to work around them. Continually brushing it under the carpet isn't going to help the situation regarding how easy Minetest is to hack/cheat. Discussing it from the point of view of "someone can do X to cheat, we can work around this by doing Y or Z" isn't endorsing/condoning/supporting cheating in anyway.

Sure, it might give some cheaters ideas about ways to cheat they wouldn't have thought of otherwise, but many of them figure it out anyway and it's necessary to discuss the vulnerability in order to figure out how to fix the vulnerability. It's not like we're posting code or instructions for producing a hacked client, if it's discussed from a technical point of view they still have to make the effort to figure out how to modify the code. Sure, modifying code is (probably) relatively easy compared to discovering the vulnerabilities/exploits themselves but if we want to make Minetest secure there's no other way to do it than by discussing the vulnerabilities in the first place.

Rant over. Here's (hopefully) to a productive discussion about adopting a security-minded approach and making Minetest more secure through open discussion of the issues and potential solutions.

[Ferret2]

Hi micheal65536,

if we want to make Minetest secure there's no other way to do it than by discussing the vulnerabilities in the first place.

Exactly!, I agree completely. Many thanks for your post.

As you said, at this moment the hackers evidently have no trouble figuring out the technical aspects of how to do the hacking, despite people wanting to keep discussion of minetest's vulnerabilites under wraps. By sweeping the hacker problem under the rug and not discussing the vulnerabilities openly, the situation is allowed to persist.

As a result, at this moment we have a game in which hackers/cheaters are given free rein, and where where honest new players joining a server inevitably will run into a hacker before long, who is completely free to ruin the game for everyone until the administrator gets online to throw him out.

I think that there will always be hackers wherever there are vulnerabilities. It is a law of nature: parasites emerge wherever there is a niche for them. If you create a system that leaves the door wide open for hackers, then inevitably hackers will come in. Since you can't fight a law of nature, therefore in my view the fault lies with the design of the client-server interaction in minetest, namely it makes life too easy on hackers.

I think the situation is similar to security in cryptography. In the course of the past 70 years or so, humanity has finally discovered that the best way to achieve good security is by discussing the problem openly. Discussing cryptography does not mean that your intent is to discover other people's secrets. It means that you want to create a system that makes it infeasible for intruders to break in. The only people who profit from keeping things under wraps are the hackers who found out for themselves how to break the system.

---
Another thought: Having discovered the hacked-client problem, I can now not help wondering how many of the regular long-time players on a server are using cheating in a subtle way that doesn't advertize clearly to the other players or the administrator, for example for finding caves and ores more quickly (and dirt pockets in JT2). Even with subtle cheating, you would become significantly more efficient in the game. I am beginning to wonder whether the real fun in online minetest is not about playing minetest, but rather about constantly balancing your cheating so that you are never caught out. I'm even wondering whether most of the long-time players find their fun in this subtle cheating, and whether this could explain why the cheating issue is not more openly addressed in the minetest community.

As long as minetest has these gaping security holes that make cheating so easy, it seems to me that every player who has discovered the hacker problem will inevitably always be wondering how much all the other players on the server are subtly cheating. I'm not the type of person who finds the idea appealing of being the only one in the game playing with a handicap, while all others have more powers (e.g. can find ores more quickly than is possible for a non-cheating player). I really detest cheating, however at the moment I am very much wondering whether if I want to keep playing minetest online, whether that then means that I am in effect forced to also cheat myself.

[Linuxdirk]

Because seriously, Minetest needs an open discussion about the technical aspects of hacking/cheating and how to work around them. Continually brushing it under the carpet isn't going to help the situation regarding how easy Minetest is to hack/cheat.

Yes, what's missing is a full disclosure mindset. Point out the cheats/glitches/hacks in maximum detail providing examples if devs do not show interest in fixing the bugs making the cheats/glitches/hacks possible. Having the dev power in mind there may be a 1 months long grace period before the full disclosure happens.

Only directly addressing issues causes attention, only attention leads to issues being solved, only solved issues prevent malicious behavior. Not directly addressing issues is endorsing malicious behavior.

[shamwow420]

Not directly addressing issues is endorsing malicious behavior.

I've long suspected there's a mole(s) within the Minetest community. My first guess as to why would be Microsoft buying the very similar Minecraft for a hefty 2 billion plus can't buy every FOSS developer but they can buy a few.

[rubenwardy]

Examples are: A player with a hacked client can give himself lots of privileges including flying and moving trough walls

This is because clients control their own movement. There's been a few attempts to resolve this, but it's hard to do and not result in lag issues. Here's one with lots of promise: https://github.com/minetest/minetest/pull/6219

and also a player with a hacked client apparently can make himself unkillable (hitpoints do not seem to go down when attacked).

This is not possible. The modified client can make them appear to not die to the player that is using it if anticheat is disabled on the server, simply by ignoring HP updates, automatically respawning, and ignoring set_pos packets. But with anticheat enabled, other players will see the modified player act as normal

In addition, just now I also saw on https://www.youtube.com/watch?v=wCK3I4HbRKw that simply by changing the texture for default stone on the client side into a transparent image, you can enable yourself to look through stone (so that you can see distant caves and the mobs and ores in them). These easily achieved hacks in a player's client installation give the cheater superpowers which make it totally impossible for normal players (who have not hacked their clients) to compete against him.

This is no longer possible with an unmodified client, as of 5.0.0.

Indeed. The general idea should be, I think, that the server should send to a client only that part of the data that the client is allowed to see. This means that the server should not send to the client exactly the data as it is represented (stored) on the server, but instead it should only send "censored" data, i.e. data from which everything is removed that that particular client is not allowed to see.

This already happens, with private node metadata

In b4 this thread gets deleted under the "no discussion of cheating" rule.

The exact text of the rule is this:

Code: Select all

Hacking/Cheating: It is not permissible to promote, link, or otherwise direct
Forum users to any kind of "hacking" or "cheating" tools, regardless of their
claimed purpose. This includes but is not limited to scripts, bots, modified
Minetest clients, CSM mods, third-party programs, etc.
Doing so is grounds for an immediate, permanent ban, and may
result deletion of all of the user's posts.


Talking about vulnerabilities and how to fix them is fine. Talking briefly about how to exploit them is fine (ie: you just need to comment out a privilege check to get a client with noclip/fly). Including working snippets or hacked client downloads is not.

This thread is perfectly fine so far. We have never deleted any discussions about vulnerabilities. The only thing this rule has been used to get rid of is: 1) prebuild hacked clients 2) bad CSM mods

Here are some threads talking about these issues:

Defeating oredetect
Guide on avoiding formspec vulnerabilities

Another thought: Having discovered the hacked-client problem, I can now not help wondering how many of the regular long-time players on a server are using cheating in a subtle way that doesn't advertize clearly to the other players or the administrator, for example for finding caves and ores more quickly (and dirt pockets in JT2). Even with subtle cheating, you would become significantly more efficient in the game. I am beginning to wonder whether the real fun in online minetest is not about playing minetest, but rather about constantly balancing your cheating so that you are never caught out. I'm even wondering whether most of the long-time players find their fun in this subtle cheating, and whether this could explain why the cheating issue is not more openly addressed in the minetest community.

I guarantee that most players do not cheat. You've basically just invented a new conspiracy theory, congrats

[DELTA_FORCE]

And let us please not forget about the "real" "hacked in" invincibility. Most players that use the standard kind of invincibility are still open to instakills. However, are we forgetting the easiest way of cheating health (and stamina and hunger, depending on the server) which is using a memory editor? Basically, the only ways to prevent most hacked clients will be to have the server to control everything, which will add much lag and doesn't seem feasible, but it is one way to solve this issue. You still can not prevent what people will do with memory editors though. (unless the server can see suspicious regeneration in the players health/stamina/hunger)

[DELTA_FORCE]

I was actually introduced to Minetest by my dad who said that Minetest would be great for using a memory editor in. Weird coincidence how it turned into this though.

[Festus1965]

I guarantee that most players do not cheat. You've basically just invented a new conspiracy theory, congrats

As some of them do it, is fact. And just one bad guy can make big damage ... what was the one guy in New Zealand causing so much pain.
I remember that our Shooter "Delta Force - Land Warrior" got really good meetings (lanparty up to 2000 gamer) and high used competitions with league AFTER we got an anticheat-addon,
so there on this lan-party have been far more gamer then I see max on Minetest ... so from this memory holding the cheaters down is important.
Just one gamer a day leaving as of this ... is too much.

Sure it might be difficult as I am not happy with anti-cheat yet, but have to keep control.

MY daughter lost really strong the fun on CTF ...

[paramat]

Point out the cheats/glitches/hacks in maximum detail providing examples if devs do not show interest in fixing the bugs making the cheats/glitches/hacks possible.
[...]
Only directly addressing issues causes attention

Nonsense.

We are always interested in fixing exploits, they are always treated as high priority.
If there is a delay it is for good reason like lack of core dev time (which is an extreme problem recently), it is not as you imply, some kind of intentional neglect.
Trying to force a solution to happen by threatening full disclosure of exploits will make no difference to this, and will just result in the exploits being used to disrupt servers.
Quite rightly, exploits are kept non-public and are privately explained to core devs, and this is what should happen.

Exploits already get attention, threats of disclosure isn't required.

[paramat]

Each player simply gets sent map blocks in which the nodes that he is not allowed to see are censored out.

it's not simple though. This is completely impractical due to how complex it is to decide what the player can be allowed to see. and the constant re-calculation and server to client updating of what the player can see as time progresses (which would also hugely increase network traffic). It would also be a lot of work to implement. It's just not worth doing even though the intention is good.

[Linuxdirk]

Nonsense.

You spelled "I personally disagree with your statement" wrong.

Quite rightly, exploits are kept non-public and are privately explained to core devs, and this is what should happen.

Yes. Full disclosure might not be the 100% correct term here. Responsible disclosure or coordinated disclosure is the better term.

But disclosure should happen after a week or a month depending on how severe the issue is and how hard it is to solve it.

The time frame results in "forcing" devs to fix the issues as soon as possible and release a fixed version instead of silently accepting the issues and hoping no-one discloses them to the public and decide to not put full effort in fixing the issue.

[rubenwardy]

This is going massively offtopic. The title for this topic contains "technical", so it would great to get it back on track

Please note that any disclosure on the forums is still subject to the "hacking/cheating" rule. Note that we fix most big security issues very quickly - 5.0.1 contains a security fix for an issue which was responsibly disclosed - however a lot of security issues like fly and noclip are hard to fix

[Linuxdirk]

Note that we fix most big security issues very quickly - 5.0.1

This is good, yes. Too bad there is roughly one to two releases a year so the fixed versions aren't rolled out to the end users (no, not everyone is using the Git versions) after the issues are fixed.

[rubenwardy]

This is good, yes. Too bad there is roughly one to two releases a year so the fixed versions aren't rolled out to the end users (no, not everyone is using the Git versions) after the issues are fixed.

We will be releasing 5.0.1 in the next few days. I'd like to do a lot more bug fixing point releases

[Linuxdirk]

We will be releasing 5.0.1 in the next few days. I'd like to do a lot more bug fixing point releases

This sounds wonderful. Let's hope all devs see it like you.

[duane]

As some of them do it, is fact. And just one bad guy can make big damage ... what was the one guy in New Zealand causing so much pain.

Can we call (some corollary of) Godwin's on this topic yet? Get some perspective, people.

I'm glad the developer's have thicker skins than I do, because if I had to take the kind of griping they do, while working for free on something that mostly benefits others, I'd walk.

[Festus1965]

I like, but I did all my knowledge already here ... what I found facts to detect and what is happen on server:
Indicators, that show NOT Minetest client on work (or a hacked one)
and as I am just starting programming in lua my strength is analyzing - other might know there software so good, that both together might find a way for solution

[sorcerykid]

For the record, here is my original response from before:

A similar concept was suggested by iisu a couple years ago. In simplest terms, it would not be a scalable client-server architecture. The engine would need to recalculate a raycast for every connected player at each server tick to determine what is viewable, while also storing mapblock state information for each client in memory and then sending mapblock state changes to every client. Minetest takes advantage of client-side prediction to lighten the load on the server and to reduce the network traffic. Realistically, what you propose would do the exact opposite in both respects.

Sorry, I don't mean to dismiss your idea wholesale, but it's too inefficient for little gain. This isn't about lackadaisical design. It's about spreading the processing load (aka distributed computing). Minetest clients needn't act as dumb terminals, when they are perfectly capable of full scene rendering and player movement :)

A more feasible way to handle cheating of this nature is to record player activity, and then flag unusually high deviations in digging times. I developed a script that analyzes the debug logs using a fairly sophisticated algorithm. It can detect with pretty high certainty what players are mining with fast, noclip, or long-range. The script first compiles the mining operations (serial digs within a given timeframe) of all players, and then applies a score based on average distance and frequency of digs involving specific valuable ores.

The only point I would add is that foolproof cheat prevention measures are always going to be a losing battle. This is the law of diminishing returns. If only a tiny fraction of players are cheating on a server, how is an excessively over-engineered anticheat implementation truly justified if all other players, as a consequence, must suffer with significantly degraded server and network performance?

Just some thoughts worth considering :)

[DELTA_FORCE]

Let's be honest. There are only a few servers where an anticheat program needs to be on, considering there isn't much a player can gain from hacking a creative server. Also, the time and effort it would take to make a quality anticheat mod that works well without costing performance would not be worth it. Also, some people who use modified clients also use a VPN, so I think the best defence against hacking is already in place (/ban, duh). What you want to do is stop people from cheating. What I am trying to say is that MT will always have backdoors and little quirks that people can use to their advantage. It will only be a matter of time before someone finds another exploit. As I said, the only way you can achieve a good anticheat is to have the server control everything (which would not be very easy, but still, you have to admit it would be the best method for cheat prevention.

[Astrobe]

The only point I would add is that foolproof cheat prevention measures are always going to be a losing battle. This is the law of diminishing returns. If only a tiny fraction of players are cheating on a server, how is an excessively over-engineered anticheat implementation truly justified if all other players, as a consequence, must suffer with significantly degraded server and network performance?

Just some thoughts worth considering :)

Agreed. To sum it up:
- automated cheat detection can be resource intensive and is subject to false negative/positives
- human-powered cheat detection is free and more reliable: you see another player floating in the air, you can see it's not because of lag or because they are using some levitation ability, so it is most likely a cheater.

The latter point also covers "exploits" made possible by mis-configured mods, mod bugs (e.g. mesecon-like mod that forgets to check for protection in a particular case) or unintended mod interactions. So it is also a lot more general.

In order to support human-powered cheat detection, one could "gamify" the hunt for cheaters, that is make it part of the game:

- Give everyone the ability to watch other players. It's not only useful against cheaters, it's also useful for helping friends and it's also just fun to watch others play too. If you want to allow some privacy, let any player watch any other player with less or same reputation as them using (shameless plug) my Reputation Points mod ( viewtopic.php?f=9&t=21964 ) for instance.

- Implement a report command that let online moderators/admins easily and quickly watch either the presumed offender or see what the reporter sees. Allow more players to play the admin role by offering semi-admin privs like temporary bans (again you could use reputation points to grant those privs automatically) or locked chests or inventory inspection, etc.

- Some mods (like the Achievements mod) probably already perform certain monitoring tasks that overlap with cheat detection, so they could be enhanced to do that too. Also, acquiring achievements very quickly can be a hint that someone is cheating. More generally, some cheaters are not particularly bright so they do fall for honey traps.

- For anti-ore-detection one could make it so that stone_with_* nodes are visually identifiable but all look the same and which ore they drops is (dig-time) random (very easy to implement). It partially defeats the cheat while introducing a different gameplay that may fit well in the context of certain games.

[micheal65536]

In b4 this thread gets deleted under the "no discussion of cheating" rule.

The exact text of the rule is this:

Code: Select all

Hacking/Cheating: It is not permissible to promote, link, or otherwise direct
Forum users to any kind of "hacking" or "cheating" tools, regardless of their
claimed purpose. This includes but is not limited to scripts, bots, modified
Minetest clients, CSM mods, third-party programs, etc.
Doing so is grounds for an immediate, permanent ban, and may
result deletion of all of the user's posts.


Talking about vulnerabilities and how to fix them is fine. Talking briefly about how to exploit them is fine (ie: you just need to comment out a privilege check to get a client with noclip/fly). Including working snippets or hacked client downloads is not.

This thread is perfectly fine so far. We have never deleted any discussions about vulnerabilities. The only thing this rule has been used to get rid of is: 1) prebuild hacked clients 2) bad CSM mods

Around the time that I discovered the 3D armor item duplication vulnerability, I mentioned that I would be providing a more thorough explanation of the vulnerability with the aim of helping other mod developers to avoid the same mistake. I also specifically stated that I would wait at least 1 month before doing so, to give server owners a chance to update to the fixed version of the 3D armor mod. I was warned, by yourself no less if I remember correctly, that this would constitute a violation of the forum rules as it would potentially be aiding cheaters in figuring out how to exploit the vulnerability (honestly though the vulnerability was simple enough that it's hard to explain or even describe without giving almost everything away).