A few months ago, when googling on realistic colour mixing using javascript, I stumbled upon a fantastic Verse.works interview to @larswander. Asked about his colour-blending approach, which is noticeable in the how you see me set of works, Lars explained some of the processes behind his artworks.

Mimi Nguyen:

So in paint-mixing, the process that takes place is subtractive - wavelengths are deleted from what we see because each paint will absorb some wavelengths that the other paint reflects. As a result, we are left with a lesser number of wavelengths remaining after the mixing process. But mixing colours physically is not the same as mixing them digitally. For digital RBG, creating a new color is by an additive color mixing process that adds one set of wavelengths to another set of wavelengths. This is based on Grassmann’s laws from 1850s in terms of algebraic equations for the additivity in the color perception of light mixtures - composed of different spectral power distributions. How did you manage to shift from paint-mixing inspiration to digital generative art?

Lars Wander:

You’re spot-on about the difference in the digital vs. physical color mixing. It also manifests itself in (subjectively) ugly ways: when mixing yellow and blue in sRGB space, you create gray, rather than green. sRGB is a natural colorspace for describing how to display digital images, but it’s pretty terrible for navigating the space between two colors. Once I realized that, I took a several-months-long journey into the rabbit hole of color spaces. That was just enough time to learn that I’m barely scratching the surface. In the process, I came across the Mixbox paper. In short, they’ve developed a way to perform Kubelka-Munk (KM) color mixing using only 3-channel sRGB colors as input. KM theory was developed in the 1930’s, and computes the spectral distribution of light reflected from a pigment mixture by physically modeling the interaction of incoming light with pigment particles. Mixbox works by decomposing each 3-channel sRGB color into synthetic cyan, magenta, yellow, and white pigment concentrations, and performing KM mixing on those resulting pigments. By using 3-channel sRGB colors as input, their software can plug into any modern digital image/paint program. Their approach is brilliant, but unfortunately, I learned after reaching out to them that they don’t have a license for indie commercial use, meaning I can’t sell any works made using Mixbox.

As many others have explained, RGB colour mixing, and particularly browser blendmodes, are very different from how pigments work in real life. This proved to be a big problem for the coding of Enfantines, since I was trying to imitate realistic brushes (crayons, markers, pencils). Time would prove that there were two sides of the problem:

• On the one hand, one has to create or find a good formula for the actual mixing of colours. For instance, if we want to add Yellow on top of Blue, the result should be Green. We need a formula that returns "GREEN" when we ask it to mix "YELLOW" and "BLUE".
• On the other hand, we need a way of performing these calculations, pixel by pixel, without heavily impacting performance —particularly if our work is animated and needs to be synchronized to music—

The Colour Mixing Formula

After reading Lars mentioning MixBox, I contacted the team and asked for an indie license. Turns out they've changed their policy and they are offering these now. Before committing to a paid license, I also went deep into colour-spaces, of course, but as Lars states, the other approaches (RYG, Subtractive, SubAdd, etc.) are unable to match the Mixbox formula's ability to predict the outcome of two pigments mixing together, where they use KM theory.

In short, I went with MixBox, for two reasons. First, I wanted the most realistic mixing algorithm possible. But more importantly, I was also looking for a simple approach where I introduced two Hex or RGB colours, a mixing ratio, and got the mixed RGB back, no need to mess with dozens of buffers per colour, and other very complicated approaches. Moreover, I needed an algorithm that was already prepared for WEBGL shaders, since I wanted to perform pixel work with the GPU for best performance.

Simply put, this is how MixBox works in Javascript:

// Import MixBox, etc.

let blue = [0, 33, 133]       // First Colour
let yellow = [252, 211, 0]    // Second Colour
let t = 0.5;                  // mixing ratio

let mixed = mixbox.lerp(blue, yellow, t);   // Resulting RGB colour

And this is how it works within a WebGL Shader:

// Import MixBox, etc.

void main(void) {

  ...

  vec3 blue = vec3(0, 0.129, 0.522);
  vec3 yellow = vec3(0.988, 0.827, 0);
    
  float t = 0.5; // mixing ratio

  vec3 mixed = mixbox_lerp(blue, yellow, t);

  ...
}

As you can see, the approach is extremely simple. You give two colours and you get the mixed colour as a result. Now, this is only a mixing formula, my problem was how to use it with p5 in an easy, scalable, and efficient way that could be useful for a wide range of applications.

Disclaimer: Yes, I'm not completely happy because this is not a fully Open Source approach and it requires a license (if you want to sell artworks that use their code). However, I'm sure there must be good mixing formulas out there, or that people really knowledgeable in colour theory will soon develop OpenSource alternatives. I went with MixBox because of a time constraint and because their algorithm is quick, efficient, and offers extremely good results. Also, I'm not being paid to write about them, on the contrary, I paid my full license (as I said, not cheap).

Important Update (Nov 2023): There is now an OpenSource alternative to Mixbox, named Spectral.js. Check it out here: https://github.com/rvanwijnen/spectral.js

At the end of this article, you can also find a simple p5 library that lets you implement realistic color-mixing on your projects, with very little friction.

The Slow Pixels[] Approach

So, we have a formula with which we can add two RGB pigments to get a realistic resulting RGB mix. Now, how do we use this? This is what we aim to do, in simple terms:

• 1 - We add Blue pigment into a circle
• 2 - We add Yellow pigment into another, intersecting circle
• 3 - The intersection part should be Green .

But really, in order for this to work, what we need to do is something like this:

• 1A - We READ current colour of every canvas pixel.
• 1B - For pixels within a circular form, we MIX Blue pigment with current colour.
• 1C - UPDATE every pixel with the mixed colour.
• 2A - We READ current colour of every canvas pixel
• 2B - For pixels within a circular form, we MIX Yellow pigment with current colour.
• 2C - UPDATE every pixel with the mixed colour.
• 3 - The intersection part is Green .

As one can see, in order to mix new pigments into the existing canvas colours, we need to read each and every pixel colour, mix the pigments for the pixels inside our geometry, and update them with the mixed RGB colour. In short, we need to do pixel work.

How to quickly perform these calculations with javascript and p5? There exist many possibilities. One of them is simply using the Pixel[] array: we loadPixels(), create a for loop where those pixels are read and, if they pass a series of checks (if they are within our circle), mix the new pigment and updatePixels(). This is the approach I used for Enfantines I, which looks and reads more or less like this:

    function circlePigment (cx,cy,radius,addColor) {
        // cx and cy are the center coordinates, radius
        // Here we calculate the left, right, top, and bottom points of the circle
        var x0 = cx-radius;
        var x1 = cx+radius;
        var y0 = cy-radius;
        var y1 = cy+radius;
    
        // This reads the pixelDensity of the display, in order to read the Pixels array without errors.
        let density = pixelDensity();
        
        // LOAD PIXELS
        loadPixels();
       
        // Start reading pixels from Top to Bottom, Left to Right.
        for (y=y0; y<y1; y++) {
            for (x=x0; x<x1; x++) {
                // Distance of pixel to Circle Center
                var distance = dist(cx, cy, x, y);    
                // If Pixel is Inside the Circle:      
                if (distance<=radius) {
                    // Adjust for High Density Displays
                    for (i = 0; i < density; i+=1) {
                    for (j = 0; j < density; j+=1) {
                          // Read p5 Documentation in order to understand this part
                          index = 4 * ((y * density + j) * width * density + (x * density + i));
                          
                          // Read Current Colour
                          var currentColor = [pixels[index],pixels[index+1],pixels[index+2]];
                          
                          // Mix Colour with MIXBOX FORMULA
                          var mixedColor = mixbox.lerp(currentColor, addColor, 0.5;
                        
                          // Upadte pixel
                          pixels[index+0] = mixedColor[0],  // R
                          pixels[index+1] = mixedColor[1],  // G
                          pixels[index+2] = mixedColor[2],  // B
                          pixels[index+3] = 255;            // Alpha
                    }
                    }
                }
            }
        }
        
        // UPDATE PIXELS
        updatePixels();
    }

However, loading pixels with our CPU and updating them, each and every-time we need to add a colour, would be very resource intensive and, probably, impossible for complex or animated artworks. In short, this is not what we need.

GPU versus CPU: Hello Shaders

Thankfully, there is a better way to perform work with Pixels: WebGL Shaders. What do Shaders Do? Better to go here and here to learn a bit about them, but:

A shader is a small program that runs entirely on your graphics card, the GPU (Graphics Processing Unit), rather than the CPU (Central Processing Unit) of your computer. This makes it incredible fast. They are actually easy to write, and simple to implement in p5, once you understand the basics of how they work.

Simply put, while the GPU performs pixel calculations "one by one", a GPU is designed to perform pixel calculations for every pixel simultaneously.

Or, best illustrated by Mythbusters:

A shader is a small program that will get executed by the GPU on each pixel. This introduces a whole set of problems which I would not dig into (particularly because I'm very new to Shaders), such as the inability to transmit information of what's happening in one pixel to the others.

Back to the topic at stake. How to perform Colour Blending with a WebGL Shader? After some digging and many tests, I've developed an approach (used in Enfantines II and, hopefully, in many more artworks in the near future) which can be potentially applied to a wide range of use-cases. It involves, of course, a shader, but a very simple shader that would not require a deep technical knowledge, very easy to implement.

But first, it's interesting to take a look at how the p5 javascript library has implemented shaders, how do we create a shader and load a shader into our sketch. In order to create a shader, we will need two parts or files, a Vert and a Frag. These can be separate files, or could be included in our same sketch.js file, as I'm doing here.

The .vert file handles everything that has to do with vertexes - that is all of your geometry (shapes) and its position on the canvas. The file ends with setting the built in variable called gl_Position equal to our calculations, this ensures that we automatically can use these positions in the .frag file.

The .frag file handles everything that has to do with the actual coloring of the pixels, and ends with setting the built in variable called gl_FragColor equal to a color. The .frag file handels "per-fragment" (per-pixel) operations. It is good practice to use this shader just for coloring of the pixels.

Quoted from: https://itp-xstory.github.io/p5js-shaders/#/./docs/how-to-write-a-shader

let colorBlendingShader;

function setup() {
    // IMPORTANT, shaders only work in WEBGL mode
    mainCanvas = createCanvas(500,500,WEBGL);
    
    // A variable that we want to send to the shader
    let newColor = [255,100,30];

    // Compile Shader
    colorBlendingShader = createShader(vert, frag);
    
    push();
    rectmode(CENTER);
    fill(0,0,0,0);
    noStroke();
    
    // Load Shader
    shader(colorBlendingShader);
    
    // Apply Shader to Canvas
    rect(0, 0, width, height);
    
    pop();
}

// Vert part of the SHADER that can be used without changes
let vert = `
precision highp float;
attribute vec3 aPosition;
attribute vec2 aTexCoord;
uniform mat4 uModelViewMatrix;
uniform mat4 uProjectionMatrix;

varying vec2 vVertTexCoord;

void main(void) {
vec4 positionVec4 = vec4(aPosition, 1.0);
  gl_Position = uProjectionMatrix * uModelViewMatrix * positionVec4;
vVertTexCoord = aTexCoord;
}
`

// Frag part of the SHADER where we will perform colour mixing
let frag = `
precision highp float;
varying vec2 vVertTexCoord;

void main() {
    
   // WHATEVER
    
}
`

The Colour Blending Shader (WebGL)

The basic idea for this shader is as follows. For each pixel, we need to have this information:

• What is its current colour?
• What colour do we want to add?
• Is it part of the geometry that we want to paint?

The first is easy. We can send our Canvas to the GPU as a texture, and we can read the colour of the current pixel with its coordinates. The second is also easy, we can send a variable as a uniform value to the Shader:

function setup() {
    // IMPORTANT, shaders only work in WEBGL mode
    mainCanvas = createCanvas(500,500,WEBGL);

    // Compile and Load Shader
    colorBlendingShader = createShader(vert, frag);
    
    push();
    fill(0,0,0,0);
    noStroke();
    rectmode(CENTER);
    
    shader(colorBlendingShader);

    // Send our CANVAS to Shader
    colorBlendingShader.setUniform('currentCanvas', mainCanvas);
    
    // Send a COLOUR to Shader
    let newPigment = [255,100,34]; // RGB values
    colorBlendingShader.setUniform('addColor', newPigment);
    
    // Apply Shader to Canvas
    rect(0, 0, width, height);
    
    pop();
}

// Frag part of the SHADER where we will perform colour mixing
let frag = `
precision highp float;
varying vec2 vVertTexCoord;

uniform sampler2D currentCanvas;  // THIS RECEIVES THE CANVAS PIXELS
uniform vec3 addColor;  // THIS RECEIVES THE NEW PIGMENT as VEC3 [Red,Green,Blue]

// THIS FUNCTION NORMALISES COLOURS FROM 0-255 values to 0-1
vec3 rgb(float r, float g, float b){
    return vec3(r / 255.0, g / 255.0, b / 255.0);
}

void main() {
    
  // THIS READS OUR CURRENT PIXEL COLOUR as a 4 VEC: [Red,Green,Blue,Alpha]
  vec4 canvasColor = texture2D(source, vVertTexCoord);
  
  // THIS is the COLOR WE WANT TO ADD, NORMALISED
  vec3 colorToAdd = rgb(addColor.r,addColor.g,addColor.b);
   
   ...
}
`

// Vert shader here... (removing for easy reading)

The third one might look easy, but it's not so easy... If we wanted to draw a circle, it would be very easy. We can send a center coordinate and a radius to the shader, and add an IF statement that adds the colour only for those pixels that are inside the circle (for which the distance to center point is lower than the radius). However, what if we want to paint a more complex shape?

At the end, I went for shortcut that allowed me keep most of my code in JavaScript, which I'm much more familiar with, and only use my shader to the colour-blending itself, a generic shader that could be used from anywhere —and by anyone, btw, that's why I'm posting it here. The approach goes like this:

Instead of drawing the geometry in the shader itself, or apply the shader to a given geometry, I created a second Buffer, aka MaskBuffer, where I will draw my geometry (ANY geometry, simple or complex) with a Red colour. Once it is drawn, we will send these buffer to the Shader as a texture (the same we do with the main Canvas), and we will only perform the colour-mixing (in the Shader) for the pixels that are RED (or have a red component).

The code would look like this —the mixBox.lutTexture() function creates an auxiliary Lookup Table (LUT) for their algorithm to work, I modified a bit their function to adapt it to p5, and will share at the end of the article—

let colorBlendingShader, mainCanvas, 

function setup() {
    // IMPORTANT, shaders only work in WEBGL mode
    mainCanvas = createCanvas(500,500,WEBGL);
    background(255,255,255)

    // Compile Shader
    colorBlendingShader = createShader(vert, frag);
    
    // Create buffer for Geometry Masks
    maskBuffer = createGraphics(width, height);
    maskBuffer.rectMode(CENTER);
    maskBuffer.noStroke();

    // MixBox auxiliary function
    mixbox.lutTexture();
    
    // Draw Geometry that will be painted
    maskBuffer.rect(width/3,height/2,300,300);
    
    push();
    fill(0,0,0,0);
    noStroke();
    rectmode(CENTER);
    
    // Load Shader
    shader(colorBlendingShader);

    // Send our CANVAS to Shader
    colorBlendingShader.setUniform('currentCanvas', mainCanvas);
    // Send a COLOUR to Shader
    let newPigment = [255,100,34]; // RGB values
    colorBlendingShader.setUniform('addColor', newPigment);
    // Send MASKBUFFER to Shader
    colorBlendingShader.setUniform('mask', maskBuffer);
    // Send Auxiliary LUT Texture to Shader
    colorBlendingShader.setUniform('mixbox_lut', mixboxTexture);
    
    // Apply Shader to Canvas
    rect(0, 0, width, height);
    
    pop();
}

// Frag part of the SHADER where we will perform colour mixing
let frag = `
precision highp float;
varying vec2 vVertTexCoord;

uniform sampler2D currentCanvas;  // THIS RECEIVES THE CANVAS PIXELS
uniform vec3 addColor;            // THIS RECEIVES THE NEW PIGMENT as VEC3 [Red,Green,Blue]

uniform sampler2D maskBuffer;     // THIS RECEIVES THE MASKBUFFER
uniform sampler2D mixbox_lut;     // THIS RECEIVES MIXBOX LUT TEXTURE

#include "mixbox.glsl"            // THIS IS MIXBOX COLOUR MIXING FORMULA


// THIS FUNCTION NORMALISES COLOURS FROM 0-255 values to 0-1
vec3 rgb(float r, float g, float b){
    return vec3(r / 255.0, g / 255.0, b / 255.0);
}

void main() {
  
  vec4 maskColor = texture2D(mask, vVertTexCoord);                         // THIS IS THE COLOUR OF THE MASK BUFFER
  vec4 canvasColor = texture2D(source, vVertTexCoord);

  vec3 colorToAdd = rgb(addColor.r,addColor.g,addColor.b);
  vec3 existingColor = vec3(canvasColor.r,canvasColor.g,canvasColor.b);    // WE NEED CANVAS AS A RGB VEC
   
  
  if (maskColor.r > 0.0) {                                             // IF PIXEL IS RED IN MASKBUFFER
        float t = 0.5;                                                 // MIX RATIO
        vec3 mixedColor = mixbox_lerp(existingColor, colorToAdd, t);   // MIX CANVAS COLOR WITH COLOR TO ADD
        gl_FragColor = vec4(mixedColor,1.0);                           // UPDATE PIXEL
    }
    else {
        gl_FragColor = vec4(existingColor,1.0);                        // ELSE, COLOR STAYS THE SAME AS IT WAS
    }
}
`

// ADD MIXBOX BLENDING FORMULA TO THE SHADER
frag = frag.replace('#include "mixbox.glsl"', mixbox.glsl());

// Vert shader here... (removing for easy reading)

However, this is extremely messy, so I've created a library that you can import to your own artwork to start using this Shader which guarantees realistic and quick colour mixing.

UPDATE: The library is now fully OpenSource, using Spectral.js, rather than mixbox.

p5.blender.js (GitHub link: https://github.com/acamposuribe/p5.blender)

Check GitHub documentation for installation and usage

CODA

I've tested this colour blending algorithm to its limits. As it stands, it's capable of executing up to 50 times per frame without impacting performance. But, why the hassle? The Image above would have been much easier if one drew three rectangles, one blue, one yellow, and one green. Well, this was developed with the idea of creating Real Paint effects such as the ones below. Enjoy the view!