Spine Unity, Color Space,Inconsistent visualization

DdzfRabbit · 24 يوليو 2023

Unity Version: 2020.3.39f1
Unity Color Space: Linear
Spine Version: 4.0.64
Spine-Unity Runtime Version: spine-unity-4.0-2022-09-26.unitypackage
Spine Color Space: Gamma

Our team has encountered a color space related issue.

When the artists in our team try to create some UI effects using slot blending modes (e.g. Addtive) in Spine, or using semi-transparent image resources. It would behave a bit differently in Spine than in Unity. We've determined that this is due to the Linear Color Space we're using in Unity, as we get consistent visual performance when trying to convert Color Space to Gamma. Additionally, I also tried to create some scenes in Unity where I tried to use Shader's Blend to implement the Addtive blend mode.

In this sample scene I used a solid color image with color values (128,128,128,255) and as you can see I used the same material for both game objects, the blend mode of this material I defined in the Shader as Blend One One to implement the Additive blend mode. When I change the color space, it has an inconsistent color mixing result.

In fact, I understand that this is an inevitable result of inconsistencies in the color space. However, our team wanted to be compatible with the workflow differences between Unity and the art at a relatively small cost and looked to the Spine team for help and a solution.

So far we've tried a "buggy" solution (but in some scenarios it works well). The way we did it was to modify the shader code in the Spine-Unity Runtime to handle the slice shader return values of the Additive shader in a wrong way. For example, we modified the Spine/SkeletonGraphic Additive so that the final color obtained does a TargetToGammaSpace operation.

This approach is definitely wrong, but algorithmically he does make the final blend result closer to what we expect, here's what we get after debugging it in our project

So our request was to get the same visual effects in Linear color space as in Spine. Unfortunately our technical director has indicated that Unity's color space cannot be easily modified, so we're stuck with this clunky approach to the problem. Would love to get help and advice from the Spine team and look forward to hearing from you.

Harald · 24 يوليو 2023

The Spine Editor provides an option to switch the viewport and any previews from gamma to linear color space, under Application - General - Color management:
http://esotericsoftware.com/spine-settings#Settings
You can set it from Gamma to Linear to get the same blending behaviour in the Spine Editor as in Unity. Did you encounter any problems with this setting, or did you perhaps not know it exists?

Unfortunately there is no precise shader-workaround available on the Unity side to get gamma-space blending when using linear color space, since combination of the src and dst colors is done outside the shaders.

DdzfRabbit · 25 يوليو 2023

Harald
Thanks for your reply, our team read the documentation carefully and noticed that the Spine editor supports previewing in linear color space since version 4.0, and tried to modify the Spine preview to Linear color space. In fact, this approach solved our problem about the inconsistency between Unity and Spine visuals, and also helped artists to preview the visual performance of Spine resources in Unity projects during the production process.

However, our art team is used to using Gamma's workflow to output art resources, so almost all of the image resources we use are sRGB compliant images, which performs well in Gamma color space, but performs poorly when switching to linear space. This is shown in the image below:

We thought we might be able to solve this problem if we could use linear image resources instead of sRGB images, unfortunately our art team is not adapted to the linear color space workflow. So we tried to convert the sRGB resource images to Linear images in PhotoShop or other image processing software. The resulting image looked like a mess, but when I imported it into the Unity project and configured the Texture correctly, it performed surprisingly well. I also took this processed image and replaced the original image resource in Spine, expecting it to solve all the problems we were having. Sadly, it didn't work as well as expected in the Spine preview. Wait, I think I'm missing something, I'll try again to see if this works.

We're not sure if other teams are experiencing this kind of workflow conflict due to color space, and look forward to some advice and help.

Harald · 25 يوليو 2023

dzfRabbit However, our art team is used to using Gamma's workflow to output art resources, so almost all of the image resources we use are sRGB compliant images

I'm not sure what exactly you mean by "sRGB compliant", but in general also when using linear color space in Unity or any game toolkit, you will still be authoring your images as normal 8 bits per channel (24bit or 32 bit) sRGB images. Linear color space does not imply that you need to be using HDR input images, except for skies and IBL maps. Linear vs gamma color space just determines in which color space blending and lighting calculations take place. Input images (your textures) and output images (your screen buffer) are still in normal sRGB color space (with a few exceptions).

dzfRabbit but performs poorly when switching to linear space. This is shown in the image below:

The problem here is likely that if the RGB channels contain high values (like 255 white or 255 blue, normalized 1.0) and the alpha channel is very transparent (e.g. normalized alpha or 0.1). Unfortunately by definition, the alpha channel is treated linearly in both gamma and linear space, which makes light transparent values brighter, while making opaque dark values darker. For example, in gamma space two additively alphablended values of blue 1.0 and alpha 0.1 would result in 1.0*0.1 + 1.0*0.1 = 0.2. In linear space, that's

toSRGB( toLinear(1.0)*0.1 + toLinear(1.0)*0.1)
= toSRGB( 1.0*0.1 + 1.0*0.1)
= toSRGB( 0.2 )  ~= 0.2 ^ (1 / 2.2)
= ~0.48

So the difference is 0.2 to 0.48 in blue intensity for the very transparent light blue case! Now the problem is that banding will occur due to spreading out previously hard-to-see values of 0-0.2 to now almost 0-0.5.

dzfRabbit We thought we might be able to solve this problem if we could use linear image resources instead of sRGB images

Unfortunately the problem is not how your input images are saved, it is how Photoshop (or whatever image editor) previews any blending and transparency for you. If your artists author art so that it looks good with gamma-space blending, it will look off with linear-space blending, and vice versa. If Photoshop only previews in gamma-space blending, then that's a problem. I would then hightly suggest to switch your Unity project to gamma space.

Unfortunately it might be difficult to author the desired blending in linear space with 32bits sRGB images, both in finding image editors that support linear blending in the first place, and even if that problem is solved, how to author highly transparent but bright parts (see below why that's a problem) and still save it as 32bit sRGB images.

dzfRabbit So we tried to convert the sRGB resource images to Linear images in PhotoShop or other image processing software.

What exactly did you do to convert to "linear images", and what format did you save it in?
If you mean that you are saving all images in .exr floating point format with 16 or 32bit per channel now, that's not recommended at all, as it would be much more expensive in terms of both memory consumption and also performance-wise.

dzfRabbit The resulting image looked like a mess, but when I imported it into the Unity project and configured the Texture correctly, it performed surprisingly well.

Did you export your image as .exr or other floating point format? By "worked surprisingly well", do you mean that the banding disappeared (which is expected), or that the values became darker as well (which would be surprising)?

dzfRabbit . I also took this processed image and replaced the original image resource in Spine, expecting it to solve all the problems we were having.

Again the question, how exactly did you "process" this image?

DdzfRabbit · 25 يوليو 2023

Harald
Thank you for your response and suggestions, I will inform my colleagues and continue to collect some questions and practices. Also I apologize for the inaccurate terminology I used in describing the problem.

As you said. I was reading from a blog post about a way to process images: modifying them in PhotoShop to 32bit per channel, and then mixing the RGB colors using grayscale coefficients. Although I'm not sure how it works. But as I mentioned before, I'm missing something. I tried to look up the log of my previous work and realized that it didn't actually behave optimally and the bands didn't disappear.

We also realized that saving image resources at 16 or 32 bit channels per channel was an undesirable way to go. Simply modifying the Shader didn't solve the problem either. We came to the same conclusion as you said - Looks good with gamma-space blending, then not so good with linear-space blending, which is almost inevitable.

We have also learned of an extremely costly approach. Uncheck sRGB (Color Texture) for all UI layer images in Unity, which will render the wrong colors. This error is then compensated for by a screen post-processing operation, and the final rendered result will look like it was represented in Gamma color space. We anticipate some performance hit and other unknown risks associated with this operation, so it is not currently used.

We get the visual effect we want in the Game view, even though it's what should have been rendered in the Scene view.

Harald · 25 يوليو 2023

dzfRabbit Also I apologize for the inaccurate terminology I used in describing the problem.

No need to apologize!

dzfRabbit I was reading from a blog post about a way to process images: modifying them in PhotoShop to 32bit per channel, and then mixing the RGB colors using grayscale coefficients.

What exactly do you mean by "mixing the RGB colors using grayscale coefficients"?
Could you perhaps share a link to this blog post? It would be interesting to read.

dzfRabbit We have also learned of an extremely costly approach. Uncheck sRGB (Color Texture) for all UI layer images in Unity, which will render the wrong colors. This error is then compensated for by a screen post-processing operation, and the final rendered result will look like it was represented in Gamma color space. We anticipate some performance hit and other unknown risks associated with this operation, so it is not currently used.

Thanks for sharing, admittedly I haven't thought of this workaround yet! I would argue this workaround of a single screen space post-processing pass should be much cheaper than using floating point exr textures at least.

The main question is however if you really want to use linear color space with cumbersome workarounds and not simply switch to gamma space instead.

DdzfRabbit · 27 يوليو 2023

Harald
Glad to have your prompt guidance!

We ultimately want to keep the workflow consistent across Unity projects and artists. Our technical director has tried to take your suggestion and switch the Unity color space to Gamma, and we're not sure if this modification is risky due to version iteration, but so far it's working well in our game.

As for the blog I mentioned earlier, unfortunately it's in the trash. I did however see something in the Adobe Support Community about Creating Linear Color space suggestion. It mentions operations related to creating documents in linear space.

Harald · 27 يوليو 2023

dzfRabbit Our technical director has tried to take your suggestion and switch the Unity color space to Gamma, and we're not sure if this modification is risky due to version iteration, but so far it's working well in our game.

That's great news, very glad to hear that! This will for sure make your lives much easier and guarantee a better result in the end.

dzfRabbit As for the blog I mentioned earlier, unfortunately it's in the trash. I did however see something in the Adobe Support Community about Creating Linear Color space suggestion. It mentions operations related to creating documents in linear space.

Thanks for sharing this blog post. At least it does not seem too complicated to switch to linear blending in Photoshop and still use 8 bits per channel. Nevertheless, the blending behaviour is still somewhat unintuitive, and the precision / banding problems are something that one can't easily bypass. Glad that you could move away from the linear color space headache.