Large schematics are spawned incompletely

[MirceaKitsune]

https://github.com/minetest/minetest/issues/8232

This is an old problem that's affected my Structures mod for years. Now that its city generator is nearing completion and I'm in the process of designing a new city with large buildings, this bug is the last thing standing in the way and posing a huge blocker to my project.

Minetest doesn't seem to properly handle spawning large schematics as part of newly generated chunks. I tried using both minetest.place_schematic and minetest.place_schematic_on_vmanip from the minetest.register_on_generated function but both come with a huge problem: The schematics are always incomplete with large cubic chunks ripped out of them. This results in every city being completely mangled to the point of structures making no sense. The bug doesn't seem to affect smaller structures at all (roughly 20 x 20 x 20 and under) even when they're being spawned rapidly, it's primarily just big schematics.

Here are a few screenshots showing this issue with the new apartment blocks I'm working on. You can see the interiors of apartments as they're randomly sliced open. Instead of the weirdly cut out cubes, we should only be seeing solid walls with windows from the outside.

[DrFrankenstone]

Is this caused by the structure being partly in chunks that hadn't been emerged when the schematic was placed? And were then generated afterwards.

You might already know if that's the cause, but if not you might be able to determine it if you create a testing world with backend = dummy in the world.mt file - that way the world is recreated every time you start and you can see how approaching the same schematic from different directions causes it to be missing from different chunks.

- If that turns out to be the problem, then workarounds should be possible.

Also, if you need a mod that shows the chunk boundaries...

Code: Select all

local data          = {} -- reuse the massive VoxelManip memory buffers instead of creating on every on_generate()

local function on_generated(minp, maxp, blockseed)

  local vm, emerge_min, emerge_max = minetest.get_mapgen_object("voxelmanip")
  vm:get_data(data)        -- put all nodes except the ground surface in this array
  local area = VoxelArea:new{MinEdge=emerge_min, MaxEdge=emerge_max}

  local nodeId_debug1 = minetest.get_content_id("default:clay")

  -- v7 rivers appear to be cutting off the top and bottom of the view area
  for x = minp.x, maxp.x do
    data[area:index(x, minp.y, minp.z)] = nodeId_debug1
    data[area:index(x, maxp.y, minp.z)] = nodeId_debug1
    data[area:index(x, minp.y, maxp.z)] = nodeId_debug1
    data[area:index(x, maxp.y, maxp.z)] = nodeId_debug1
  end
  for y = minp.y, maxp.y do
    data[area:index(minp.x, y, minp.z)] = nodeId_debug1
    data[area:index(maxp.x, y, minp.z)] = nodeId_debug1
    data[area:index(minp.x, y, maxp.z)] = nodeId_debug1
    data[area:index(maxp.x, y, maxp.z)] = nodeId_debug1
  end
  for z = minp.z, maxp.z do
    data[area:index(minp.x, minp.y, z)] = nodeId_debug1
    data[area:index(maxp.x, minp.y, z)] = nodeId_debug1
    data[area:index(minp.x, maxp.y, z)] = nodeId_debug1
    data[area:index(maxp.x, maxp.y, z)] = nodeId_debug1
  end

  vm:set_data(data)   
  vm:set_lighting({day=0, night=0})
  vm:calc_lighting()
  vm:write_to_map()
end

minetest.register_on_generated(on_generated)


[MirceaKitsune]

I should clarify that I'm already using the vmanip to spawn structures with on_generate. I understand this function should now only attempt to spawn a piece of schematic that intersects the chunk in cause. So this shouldn't be a case of chunk overflow normally, since I use the vm provided by the generator which should know what to do.

That is an interesting test, I didn't know about this parameter. I will keep in mind.

[DrFrankenstone]

FWIW, I didn't mean that drawing beyond the bounds of the chunk might cause it, I meant that being limited to within the bounds of the on_generate chunk may cause it if you don't also successfully spawn the structure in every other chunk that the structure touches.

Are the missing parts aligned with chunk boundaries?

[MirceaKitsune]

FWIW, I didn't mean that drawing beyond the bounds of the chunk might cause it, I meant that being limited to within the bounds of the on_generate chunk may cause it if you don't also successfully spawn the structure in every other chunk that the structure touches.

Are the missing parts aligned with chunk boundaries?

Oh: I am calling minetest.place_schematic_on_vmanip for the same structure multiple times, in case it intersects multiple chunks. Each minetest.on_generated call should be addressing every remaining piece of a schematic.

It doesn't seem like they're aligned. The large cubic chunks ripped out of buildings appear to be random and their size seems to vary. I still suspect it has something to do with chunks but we need to dig deeper.

[DrFrankenstone]

Yeah, I've seen it a couple of times now too, and it doesn't seem to be aligned with chunk boundaries. It also appears to be deterministic - reappearing every time the map is generated, even with the backend set to dummy.

Edit: the position of the player when the map is being generated can sometimes affect where holes appear.
(player position changes the order chunks are generated in)

[MirceaKitsune]

Very interesting. Thank you for testing and mentioning those details, it may help in finding a solution.

[Sokomine]

I had some strange behaviour with my skyplatform mod as well. Apart from the roads and the dirt with grass in the middle, there ought to be only sand as ground. Each platform is 100x100 nodes wide and placed using minetest.place_schematic(..):

[DrFrankenstone]

I suspect a concurrency bug.

I've been using large schematics to create giant trees. The replacement table passed to place_schematic() is different for each type of tree, and I've noticed that if two trees are near each other (like in the cherry blossom screenshot), the first replacement table ends up being applied to both schematics, even though a very different table was passed to the second place_schematic() call.

My thinkings is that the replacement table might not be the only thing inadvertently being shared by invocations of place_schematic() (which might be running in parallel).

[Sokomine]

I've been using large schematics to create giant trees. The replacement table passed to place_schematic() is different for each type of tree, and I've noticed that if two trees are near each other (like in the cherry blossom screenshot), the first replacement table ends up being applied to both schematics, even though a very different table was passed to the second place_schematic() call.

The replacement table is definitely cached. When I noticed that some time ago, I made sure lua_api.txt was adjusted to mention that behaviour.

My thinkings is that the replacement table might not be the only thing inadvertently being shared by invocations of place_schematic() (which might be running in parallel).

Possibly. Borders between mapchunks seem to cause some problems. Perhaps it's really mapgen running in parallel that causes it. Except that in the example above the sourrounding mapchunks ought to have long been created.

[DrFrankenstone]

damn, I saw that in the lua_api.txt (thank you) and forgot. Working around this is going to be interesting.

[texmex]

Does schemlib solve the issue at mapchunk borders? IIRC it dealt with such problems.

[paramat]

What value are you using for 'num_emerge_threads'? The default is automatic behaviour which is 'number of processors' - 2. Also note that a sqlite world database is not 'thread safe' so 'num_emerge_threads' should be set to '1'.
Try '1' to see if this improves anything.

[DrFrankenstone]

What value are you using for 'num_emerge_threads'? The default is automatic behaviour which is 'number of processors' - 2. Also note that a sqlite world database is not 'thread safe' so 'num_emerge_threads' should be set to '1'.
Try '1' to see if this improves anything.

I just tried setting to to 1 (it was 0), and that hasn't fixed it.

The pieces that go missing seem to be 16x16, though usually they are bunched together as bigger missing pieces.

[rubenwardy]

I had this bug in CTF, and found that you need to emerge the area first using minetest.emerge_area

[MirceaKitsune]

Confirming that num_emerge_threads = 1 in minetest.conf does not seem to affect the problem, it still occurs the same way.

[DrFrankenstone]

I had this bug in CTF, and found that you need to emerge the area first using minetest.emerge_area

My lua code's in the on_generate, so partly is the emerge code, but it does improve if I move the place_schematic() call inside a minetest.after(0.1, ...) call to help it happen after the emerge.

[bell07]

You can try to use the emerge_area hook like schemlib does, but in combination with minetest.place_schematic:
(untested code based on schemlib code:)

Code: Select all

local function emergeblocks_callback(pos, action, num_calls_remaining, ctx)
   if not ctx.total_blocks then
      ctx.total_blocks   = num_calls_remaining + 1
      ctx.current_blocks = 0
   end
   ctx.current_blocks = ctx.current_blocks + 1

   if ctx.current_blocks == ctx.total_blocks then
      minetest.place_schematic(ctx.pos, ctx.schematic, ctx.rotation, ctx.replacements, ctx.force_placement)
   end
end


local function my_place_schematic((pos, schematic, rotation, replacements, force_placement))
   minetest.emerge_area(pos, pos, emergeblocks_callback, {
      pos = pos,
      schematic = schematic,
      rotation = rotation,
      replacements = replacements,
      force_placement = force_placement
   })
end