Building Sediment Part 2: Visuals
We're back with part 2 of our technical breakdown of sediment, this time focusing on the visuals, you can find part 1 here: Building Sediment Part 1: Gameplay Systems
The look of sediment evolved radically over the course of development, while we had a wealth of concept art showing it in different forms, from creatures to ships, and many references giving inspiration for its different states, we didn't have exact key art we needed to recreate. Part of this was due to not knowing what was even possible for such a dynamic material in a real-time game environment, and part was intentional, to allow any interesting ideas from experimenting with rendering techniques to bring about something new and exciting.
Ultimately we wanted sediment to have a fractured surface, that when broken down, revealed a detailed structure of aligned rods, the patterns and structures echoing long forgotten architecture from the Monolith’s early planet based civilisation.
Our experiments ranged from instancing geometry over surfaces, generating meshes and applying subdivision effects at runtime, real-time constructive solid geometry, ray marching, and finally the instancing approach shown here.
There are many layers to the effect, but the core geometry is created by instancing pre-fractured meshes inside the fill area, then scaling and warping their fracture pieces to conform to the shape and add surface detail.
Following are the steps we take to render…
Copy voxel fill data from the CPU array to a 3D texture
Extrapolate border voxels to improve accuracy of the next step
Create a signed distance field (SDF) representing the fill boundary
Instance meshes in filled areas, using pre-fractured meshes near the fill boundary
Hide pieces that are in empty areas, and warp them conform to the SDF
Rotate pieces to create a fractured surface, and adjust colour near the melted area
Generating an SDF
Debug view of a partially melted sediment chunk, showing the signed distance field contours
From the voxel data, we get an implicit surface where fill values of 1 are inside, 0 are outside, and 0.5 can be considered on the boundary. We tried generating geometry that matched this surface, however the blocky nature of the data created a lot of artefacts as a chunk melted and the boundary moved, particularly noticeable when we had a low-resolution voxel grid combined with more detailed geometry. This was quite a common scenario as we wanted to keep the two decoupled, so we could balance CPU and GPU resources, and adjust the visuals without breaking carefully tuned gameplay.
Signed distance fields (SDFs) are anything that tell you how far a given position is from the surface of some geometry, positive if you are outside the geometry, negative if you are inside. By processing our fill values and turning them into an accurate SDF, we were able to smooth out discontinuities in the data, and unlock a lot of useful techniques.
For runtime speed, we initially tried using the MIPs of the voxel fill 3D texture as an approximate SDF, but picking a high LOD means you lose detail close to the boundary, and a low LOD doesn’t tell you anything far from the boundary. There is probably a solution based on blending LODs, but we didn’t pursue this.
We considered the popular jump flood algorithm (JFA), which can extract an accurate SDF after a few iterations, but it’s not very practical for our use case. We’re working with 3D textures, so we’d need a 3-channel texture; we want inside and outside distances so we’d need to run it twice; and the resolution of our data would require running the algorithm for many iterations per-frame.
After exploring these, we settled on a simpler flood-fill approach. Melting is quite slow, so the SDF does not need to change drastically from frame to frame, allowing us to reuse data from previous frames and update at a slower rate. We focus our effort on making sure the boundary responds quickly to the player, and update areas further away a few frames later.
This worked out much cheaper than JFA in both computation and memory cost, requiring only a single channel texture to work, we had enough accuracy in 16 bits to encode a distance of 8 metres. Ultimately we ended up generating 4 separate SDFs for various effects in parallel using the 4 channels of an RGBA texture.
Here’s some example code in 2D, the game does the same but in 3D https://www.shadertoy.com/view/NslfWs
Using the same texture size as our CPU voxel data creates some problems at the edges. Our fill data is representative of voxel centres, however our actual geometry needs to fill the entire voxel, corner to corner. For voxels inside the volume, we can get values for corners by interpolating between the centre values of neighbouring voxels, this is done for us by the hardware’s bilinear texture filtering. For voxels on the edge of the volume, outer corners have no neighbours to interpolate, so the value is clamped, resulting in an inaccurate SDF at the volume edges. We fix this by adding a border of voxels around the original volume and extrapolating their fill values before calculating the SDF.
Placing instances
Pre-fractured instance meshes
A chunk with the corner melted, just showing instance placement, more detailed meshes are placed near the melted area
We want fill the volume with mesh instances, with a few constraints:
High-poly meshes with extra fracturing are placed near the fill area boundary
Low-poly meshes are placed inside the fill area
We don’t place meshes outside the fill area
The SDF helps when working out where instances are, relative to the boundary. Sampling an SDF at a point can tell us if a sphere centred at that point is inside, outside, or on the boundary. Our meshes are rectangular, so we approximate their shape with two bounding spheres:
This step is done in compute, which writes out lists of transforms to buffers used by the instanced draw call later.
Transforming mesh pieces
Transformed mesh pieces during melting
Transformed mesh pieces while at rest
Each pre-fractured piece of the mesh needs to be transformed to alter the surface appearance:
Scaling and rotating at random to add roughness and fissure lines
Shrinking pieces on the melt boundary, so they appear to erode
Hiding pieces outside the boundary
Twisting pieces near the melt boundary, so the chunk appears to break apart
A list of transforms, one for each mesh piece of each instance, is written to a buffer and then applied later in the vertex stage.
Initially this was calculated in the vertex stage, which made iteration quick, but became a performance bottleneck. Moving the work to compute, so it could be done per-piece instead of per-vertex, reduced the load by 48x (each piece has 24 vertices, and Unity runs the vertex stage twice when you have a depth prepass enabled). This also allows us to only run this stage while the chunk is being melted, and persist the results for later frames when it’s static.
Smoothing
Applying simple scaling and rotation to the mesh pieces leaves us with a blocky appearance, which we used for quite some time, until the art team started asking how to create curved chunks...
Calculating the gradient of the SDF (the direction to the boundary) gives us a useful tool to achieve this. Given a vertex position, we can look up the direction and distance to the boundary, then move our vertex to that closest point on the boundary. Doing this for all vertices outside the boundary makes the geometry match the SDF. The normal of the triangle surface still represents the old position so shading will still have hard edges, but this is easily fixed by replacing the surface normal with the SDF gradient.
When melting, there's a lot of interesting animation detail that we wanted to preserve, however it was slightly too noisy, and we need the surface to appear as if it’s liquefying. We blend just a little of the SDF gradient into the surface normal to smooth it out, and combine with some sinusoidal vertex motion:
Melting state, before pixel smoothing
Melting state, after pixel smoothing
When at rest, the sediment heals into a smooth hard surface with a few jagged pieces poking out, indicating that it's been disturbed. This is done by randomly selecting a few pieces to remain untouched, and repositioning vertices and replacing vertex normals of the rest:
Rest state, before vertex smoothing
Rest state, after vertex smoothing
Surface shading
Fissure glow
Sediment has an inner glow that seeps out through fissures in the surface. We mask out fissures by comparing the vertex normal with the SDF gradient; wherever they are perpendicular, we add some emission to the surface:
Bevels
To help pick out detail in the surface from glancing light, we add a slight bevel to fissured areas. The thickness of the bevel is adjusted depending on how fractured or melted a mesh piece is, so the normal is calculated in shader, rather than using a normal map.
It’s quite a simple function, taking the face uv, the size of the face, and the bevel thickness we desire https://www.shadertoy.com/view/7ttGWS
Without bevels
With bevels
Instancing and batching
We use instancing to improve the performance of drawing many individual meshes that make up a chunk, but sometimes we have many separate chunks. Separate chunks do not share GPU resources, they use their own data textures and buffers, so they cannot be easily instanced and require separate draw calls. The CPU cost of doing these draw calls really adds up.
To fix this, we bake out static versions of the geometry that don't use any of the dynamic textures or buffers, allowing Unity’s SRP batcher to combine them into optimised draw calls. These static meshes are swapped with the dynamic system whenever the camera gets close enough, or if the player starts interacting with the chunk.
Baking is done in compute at edit time, which generates triangles by taking the instancing and mesh data, and running the same vertex stage transformations against them.
Reality melting
A fairly late addition to the system was the reality melting effect seen in the last few levels. This uses the SDF to blend between a static environment mesh (using alpha testing), and a sediment chunk that’s sculpted to match the geometry.