Large schematics are spawned incompletely

Large schematics are spawned incompletely

Postby MirceaKitsune » Thu Feb 14, 2019 8:50 pm

https://github.com/minetest/minetest/issues/8232

This is an old problem that's affected my Structures mod for years. Now that its city generator is nearing completion and I'm in the process of designing a new city with large buildings, this bug is the last thing standing in the way and posing a huge blocker to my project.

Minetest doesn't seem to properly handle spawning large schematics as part of newly generated chunks. I tried using both minetest.place_schematic and minetest.place_schematic_on_vmanip from the minetest.register_on_generated function but both come with a huge problem: The schematics are always incomplete with large cubic chunks ripped out of them. This results in every city being completely mangled to the point of structures making no sense. The bug doesn't seem to affect smaller structures at all (roughly 20 x 20 x 20 and under) even when they're being spawned rapidly, it's primarily just big schematics.

Here are a few screenshots showing this issue with the new apartment blocks I'm working on. You can see the interiors of apartments as they're randomly sliced open. Instead of the weirdly cut out cubes, we should only be seeing solid walls with windows from the outside.

Image
Image
Image
Image
MirceaKitsune
Member
 
Posts: 851
Joined: Sat May 21, 2011 10:31 pm
GitHub: MirceaKitsune
In-game: MirceaKitsune

Re: Large schematics are spawned incompletely

Postby DrFrankenstone » Sat Feb 16, 2019 9:11 am

Is this caused by the structure being partly in chunks that hadn't been emerged when the schematic was placed? And were then generated afterwards.

You might already know if that's the cause, but if not you might be able to determine it if you create a testing world with backend = dummy in the world.mt file - that way the world is recreated every time you start and you can see how approaching the same schematic from different directions causes it to be missing from different chunks.

- If that turns out to be the problem, then workarounds should be possible.

Also, if you need a mod that shows the chunk boundaries...
Code: Select all
local data          = {} -- reuse the massive VoxelManip memory buffers instead of creating on every on_generate()

local function on_generated(minp, maxp, blockseed)

  local vm, emerge_min, emerge_max = minetest.get_mapgen_object("voxelmanip")
  vm:get_data(data)        -- put all nodes except the ground surface in this array
  local area = VoxelArea:new{MinEdge=emerge_min, MaxEdge=emerge_max}

  local nodeId_debug1 = minetest.get_content_id("default:clay")

  -- v7 rivers appear to be cutting off the top and bottom of the view area
  for x = minp.x, maxp.x do
    data[area:index(x, minp.y, minp.z)] = nodeId_debug1
    data[area:index(x, maxp.y, minp.z)] = nodeId_debug1
    data[area:index(x, minp.y, maxp.z)] = nodeId_debug1
    data[area:index(x, maxp.y, maxp.z)] = nodeId_debug1
  end
  for y = minp.y, maxp.y do
    data[area:index(minp.x, y, minp.z)] = nodeId_debug1
    data[area:index(maxp.x, y, minp.z)] = nodeId_debug1
    data[area:index(minp.x, y, maxp.z)] = nodeId_debug1
    data[area:index(maxp.x, y, maxp.z)] = nodeId_debug1
  end
  for z = minp.z, maxp.z do
    data[area:index(minp.x, minp.y, z)] = nodeId_debug1
    data[area:index(maxp.x, minp.y, z)] = nodeId_debug1
    data[area:index(minp.x, maxp.y, z)] = nodeId_debug1
    data[area:index(maxp.x, maxp.y, z)] = nodeId_debug1
  end

  vm:set_data(data)   
  vm:set_lighting({day=0, night=0})
  vm:calc_lighting()
  vm:write_to_map()
end

minetest.register_on_generated(on_generated)
DrFrankenstone
Member
 
Posts: 103
Joined: Tue May 24, 2016 5:36 am
GitHub: treer

Re: Large schematics are spawned incompletely

Postby MirceaKitsune » Sat Feb 16, 2019 11:51 am

I should clarify that I'm already using the vmanip to spawn structures with on_generate. I understand this function should now only attempt to spawn a piece of schematic that intersects the chunk in cause. So this shouldn't be a case of chunk overflow normally, since I use the vm provided by the generator which should know what to do.

That is an interesting test, I didn't know about this parameter. I will keep in mind.
MirceaKitsune
Member
 
Posts: 851
Joined: Sat May 21, 2011 10:31 pm
GitHub: MirceaKitsune
In-game: MirceaKitsune

Re: Large schematics are spawned incompletely

Postby DrFrankenstone » Sat Feb 16, 2019 12:07 pm

FWIW, I didn't mean that drawing beyond the bounds of the chunk might cause it, I meant that being limited to within the bounds of the on_generate chunk may cause it if you don't also successfully spawn the structure in every other chunk that the structure touches.

Are the missing parts aligned with chunk boundaries?
DrFrankenstone
Member
 
Posts: 103
Joined: Tue May 24, 2016 5:36 am
GitHub: treer

Re: Large schematics are spawned incompletely

Postby MirceaKitsune » Sat Feb 16, 2019 12:14 pm

DrFrankenstone wrote:FWIW, I didn't mean that drawing beyond the bounds of the chunk might cause it, I meant that being limited to within the bounds of the on_generate chunk may cause it if you don't also successfully spawn the structure in every other chunk that the structure touches.

Are the missing parts aligned with chunk boundaries?


Oh: I am calling minetest.place_schematic_on_vmanip for the same structure multiple times, in case it intersects multiple chunks. Each minetest.on_generated call should be addressing every remaining piece of a schematic.

It doesn't seem like they're aligned. The large cubic chunks ripped out of buildings appear to be random and their size seems to vary. I still suspect it has something to do with chunks but we need to dig deeper.
MirceaKitsune
Member
 
Posts: 851
Joined: Sat May 21, 2011 10:31 pm
GitHub: MirceaKitsune
In-game: MirceaKitsune

Re: Large schematics are spawned incompletely

Postby DrFrankenstone » Sun Feb 17, 2019 3:57 am

Yeah, I've seen it a couple of times now too, and it doesn't seem to be aligned with chunk boundaries. It also appears to be deterministic - reappearing every time the map is generated, even with the backend set to dummy.

Edit: the position of the player when the map is being generated can sometimes affect where holes appear.
(player position changes the order chunks are generated in)
DrFrankenstone
Member
 
Posts: 103
Joined: Tue May 24, 2016 5:36 am
GitHub: treer

Re: Large schematics are spawned incompletely

Postby MirceaKitsune » Sun Feb 17, 2019 1:26 pm

Very interesting. Thank you for testing and mentioning those details, it may help in finding a solution.
MirceaKitsune
Member
 
Posts: 851
Joined: Sat May 21, 2011 10:31 pm
GitHub: MirceaKitsune
In-game: MirceaKitsune

Re: Large schematics are spawned incompletely

Postby Sokomine » Wed Feb 20, 2019 12:48 pm

I had some strange behaviour with my skyplatform mod as well. Apart from the roads and the dirt with grass in the middle, there ought to be only sand as ground. Each platform is 100x100 nodes wide and placed using minetest.place_schematic(..):
Image
Sokomine
Member
 
Posts: 3750
Joined: Sun Sep 09, 2012 5:31 pm
GitHub: Sokomine

Re: Large schematics are spawned incompletely

Postby DrFrankenstone » Sun Feb 24, 2019 1:47 pm

I suspect a concurrency bug.

I've been using large schematics to create giant trees. The replacement table passed to place_schematic() is different for each type of tree, and I've noticed that if two trees are near each other (like in the cherry blossom screenshot), the first replacement table ends up being applied to both schematics, even though a very different table was passed to the second place_schematic() call.

My thinkings is that the replacement table might not be the only thing inadvertently being shared by invocations of place_schematic() (which might be running in parallel).
DrFrankenstone
Member
 
Posts: 103
Joined: Tue May 24, 2016 5:36 am
GitHub: treer

Re: Large schematics are spawned incompletely

Postby Sokomine » Sun Feb 24, 2019 2:02 pm

DrFrankenstone wrote:I've been using large schematics to create giant trees. The replacement table passed to place_schematic() is different for each type of tree, and I've noticed that if two trees are near each other (like in the cherry blossom screenshot), the first replacement table ends up being applied to both schematics, even though a very different table was passed to the second place_schematic() call.

The replacement table is definitely cached. When I noticed that some time ago, I made sure lua_api.txt was adjusted to mention that behaviour.

DrFrankenstone wrote:My thinkings is that the replacement table might not be the only thing inadvertently being shared by invocations of place_schematic() (which might be running in parallel).

Possibly. Borders between mapchunks seem to cause some problems. Perhaps it's really mapgen running in parallel that causes it. Except that in the example above the sourrounding mapchunks ought to have long been created.
Sokomine
Member
 
Posts: 3750
Joined: Sun Sep 09, 2012 5:31 pm
GitHub: Sokomine

Re: Large schematics are spawned incompletely

Postby DrFrankenstone » Sun Feb 24, 2019 2:25 pm

damn, I saw that in the lua_api.txt (thank you) and forgot. Working around this is going to be interesting.
DrFrankenstone
Member
 
Posts: 103
Joined: Tue May 24, 2016 5:36 am
GitHub: treer

Re: Large schematics are spawned incompletely

Postby texmex » Sun Feb 24, 2019 7:01 pm

Does schemlib solve the issue at mapchunk borders? IIRC it dealt with such problems.
texmex
Member
 
Posts: 1290
Joined: Mon Jul 11, 2016 9:08 pm
GitHub: tacotexmex
In-game: texmex

Re: Large schematics are spawned incompletely

Postby paramat » Sat Mar 02, 2019 1:31 am

What value are you using for 'num_emerge_threads'? The default is automatic behaviour which is 'number of processors' - 2. Also note that a sqlite world database is not 'thread safe' so 'num_emerge_threads' should be set to '1'.
Try '1' to see if this improves anything.
paramat
Developer
 
Posts: 3364
Joined: Sun Oct 28, 2012 12:05 am
GitHub: paramat

Re: Large schematics are spawned incompletely

Postby DrFrankenstone » Sat Mar 02, 2019 1:53 am

paramat wrote:What value are you using for 'num_emerge_threads'? The default is automatic behaviour which is 'number of processors' - 2. Also note that a sqlite world database is not 'thread safe' so 'num_emerge_threads' should be set to '1'.
Try '1' to see if this improves anything.


I just tried setting to to 1 (it was 0), and that hasn't fixed it.

The pieces that go missing seem to be 16x16, though usually they are bunched together as bigger missing pieces.
DrFrankenstone
Member
 
Posts: 103
Joined: Tue May 24, 2016 5:36 am
GitHub: treer

Re: Large schematics are spawned incompletely

Postby rubenwardy » Sat Mar 02, 2019 12:54 pm

I had this bug in CTF, and found that you need to emerge the area first using minetest.emerge_area
rubenwardy
Moderator
 
Posts: 5725
Joined: Tue Jun 12, 2012 6:11 pm
GitHub: rubenwardy
In-game: rubenwardy

Re: Large schematics are spawned incompletely

Postby MirceaKitsune » Sat Mar 02, 2019 10:43 pm

Confirming that num_emerge_threads = 1 in minetest.conf does not seem to affect the problem, it still occurs the same way.
MirceaKitsune
Member
 
Posts: 851
Joined: Sat May 21, 2011 10:31 pm
GitHub: MirceaKitsune
In-game: MirceaKitsune

Re: Large schematics are spawned incompletely

Postby DrFrankenstone » Sat Mar 02, 2019 11:32 pm

rubenwardy wrote:I had this bug in CTF, and found that you need to emerge the area first using minetest.emerge_area


My lua code's in the on_generate, so partly is the emerge code, but it does improve if I move the place_schematic() call inside a minetest.after(0.1, ...) call to help it happen after the emerge.
DrFrankenstone
Member
 
Posts: 103
Joined: Tue May 24, 2016 5:36 am
GitHub: treer

Re: Large schematics are spawned incompletely

Postby bell07 » Wed Apr 03, 2019 5:41 am

You can try to use the emerge_area hook like schemlib does, but in combination with minetest.place_schematic:
(untested code based on schemlib code:)
Code: Select all
local function emergeblocks_callback(pos, action, num_calls_remaining, ctx)
   if not ctx.total_blocks then
      ctx.total_blocks   = num_calls_remaining + 1
      ctx.current_blocks = 0
   end
   ctx.current_blocks = ctx.current_blocks + 1

   if ctx.current_blocks == ctx.total_blocks then
      minetest.place_schematic(ctx.pos, ctx.schematic, ctx.rotation, ctx.replacements, ctx.force_placement)
   end
end


local function my_place_schematic((pos, schematic, rotation, replacements, force_placement))
   minetest.emerge_area(pos, pos, emergeblocks_callback, {
      pos = pos,
      schematic = schematic,
      rotation = rotation,
      replacements = replacements,
      force_placement = force_placement
   })
end
bell07
Member
 
Posts: 561
Joined: Sun Sep 04, 2016 3:15 pm
GitHub: bell07



Return to Problems



Who is online

Users browsing this forum: Bing Bot [Bot] and 0 guests