#DIYConsole – Part 3: Screen buffer and pixel drawing

In the previous post we started writing the #DIYConsole graphics library, beginning from the initialization of the hardware. In this post we will finally begin to draw something on the screen!

Among other things, we have seen that the display is connected to the ESP32 over SPI and that a simple communication protocol is used, which involves sending a command byte followed by the related parameters (if any).

How to draw on the screen

To draw on the screen the procedure is as follows:

1. Set the address window: the address window is the area of the screen where the pixels will be drawn when we send image data. The pixels are drawn inside this area starting from the top left corner and proceeding from left to right and from top to bottom. The commands used to set the address window are CASET (Column Address Set) and PASET (Page Address Set).
2. Send RAMWR (Memory Write) command: this command is used to enable the transfer of data from MCU to the video memory of the display.
3. Send the image data: the pixels are sent in the order seen above; each pixel is made up of 2 bytes (16 bit color depth).

When we want to update the image on the screen, we could send these commands for every change we want to make to the image. However, writing directly into the display video memory may result in flickering problems, i.e. unintended display of intermediate images for a short time. For example, drawing a game screen by blanking an area first, then drawing sprites on top of it, makes it possible for the blank region to appear momentarily on screen.

To avoid flicker, we need to use a screen buffer, a portion of RAM where we can draw off-screen the next frame prior to showing it. In this way the whole frame is sent to video memory in a single pass without intermediate images. Although this technique greatly reduces flicker, it can be very inefficient. We will see in a future post how we can optimize it to get better frame rates.

The screen buffer

We must allocate a screen buffer big enough to contain an entire frame. Since the screen resolution is 320×480, and the color depth is 16 bits (2 bytes), the total size of the buffer should be 320x480x2 = 307.200 bytes (300 KiB). It may not seem like much, as we’re used to think in terms of gigabytes of RAM, but in the embedded world it’s a huge amount of memory! The ESP32 has 520 KiB of RAM, and only 320 KiB are used for data (approximately 290 KiB are actually available to the user).

So, a screen buffer of this size is out of the question. To reduce the size of the screen buffer we have two ways: either we reduce the resolution or the color depth. The simplest solution, which still leaves a certain degree of flexibility, is to switch to a color depth of 8 bits and use this byte as an index for a Color Lookup Table (CLUT, more commonly called a palette) which will store the colors that will be drawn on the screen. The size of the screen buffer is then halved to 153.600 bytes. Still big, but at least within the available memory.

Screen buffer memory allocation

We can allocate the screen buffer using the C library function malloc. Let’s try this code and see what happens:

void setup() {
    Serial.begin(115200);
    uint8_t* ptr = (uint8_t*)malloc(153600);
    if(ptr == NULL)
        Serial.println("Malloc failed!");
}

void loop() {}

Oops, it doesn’t work! Why? Try this other code:

void setup() {
    Serial.begin(115200);
    Serial.println(heap_caps_get_largest_free_block(MALLOC_CAP_8BIT));
}

void loop() {}

The function heap_caps_get_largest_free_block is part of the ESP-IDF (Espressif IoT Development Framework) and returns the biggest block you can allocate. If you run this code, you will see that the largest block you can allocate is smaller than we need (it returns 113792 bytes).

As a result, we are forced to split the screen buffer into two sectors. I decided to divide it like this: a buffer for the top part of the screen with a size of 320×256 pixels, and a buffer for the remaining 320×224 pixels of the bottom part. I chose a height of 256 pixels for the upper buffer because using a power of 2 will allow us in future to use some “tricks” to optimize the code.

Let’s start to update our graphics library. First, we need to declare the two screen buffer sectors. Add this code to the GFX class in the GFX.h file:

private:
uint8_t* screenBuffer[2]; // 0 => Top sector, 1 => Bottom sector

Then, at the end of the begin method, add the allocation code:

// Allocate framebuffer
screenBuffer[0] = (uint8_t*)malloc(81920); // 320x256 pixels
screenBuffer[1] = (uint8_t*)malloc(71680); // 320x224 pixels

The palette

As we saw earlier, the screen buffer does not contain the color to display on the screen, but an index to be used with a palette. So we have to add the declaration of the palette array in the private section of the GFX class:

uint16_t palette[256];

The color format used by the display is RGB565. In this format, the color components are packed in 16 bits as follows: 5 bits are used for red, 6 bits for green and 5 for blue. Interesting fact: green has an extra bit because the human eye is more sensitive to light in that part of the spectrum.

Since we are used to work with the classic 24 bit per pixel format, it is convenient to have a macro that converts this format to RGB565. Add this line to GFX.h:

#define RGB565(r,g,b) (((r & 0xF8) << 8) | ((g & 0xFC) << 3) | ((b & 0xF8) >> 3))

Now that we have declared the palette array, we need a method to load a palette in memory. In addition, we will also write a method to load a default 16 colors palette.

the 16 colors default palette of the #DIYConsole

Add these public declarations to GFX.h:

void loadDefaultPalette();
void loadPalette(uint16_t* newPalette, int size);

and the definitions in GFX.cpp:

void GFX::loadDefaultPalette() {
    memset(palette, 0, 512);
    palette[0] =  RGB565(0xFF, 0xFF, 0xFF); // White
    palette[1] =  RGB565(0xFF, 0xFF, 0x00); // Yellow
    palette[2] =  RGB565(0xFF, 0xA5, 0x00); // Orange
    palette[3] =  RGB565(0xFF, 0x00, 0x00); // Red
    palette[4] =  RGB565(0xEE, 0x82, 0xEE); // Violet
    palette[5] =  RGB565(0x4B, 0x00, 0x82); // Indigo
    palette[6] =  RGB565(0x00, 0x00, 0xFF); // Blue
    palette[7] =  RGB565(0x00, 0xBF, 0xFF); // DeepSkyBlue
    palette[8] =  RGB565(0x32, 0xCD, 0x32); // LimeGreen
    palette[9] =  RGB565(0x00, 0x64, 0x00); // DarkGreen
    palette[10] = RGB565(0x8B, 0x45, 0x13); // SaddleBrown
    palette[11] = RGB565(0xD2, 0xB4, 0x8C); // Tan
    palette[12] = RGB565(0xD3, 0xD3, 0xD3); // LightGray
    palette[13] = RGB565(0xA9, 0xA9, 0xA9); // DarkGray
    palette[14] = RGB565(0x69, 0x69, 0x69); // DimGray
    palette[15] = RGB565(0x00, 0x00, 0x00); // Black
}

void GFX::loadPalette(uint16_t* newPalette, int size) {
    memset(palette, 0, 512);
    memcpy(palette, newPalette, 2 * size);
}

Drawing pixels on the screen buffer

The manipulation of the screen buffer is the main activity of any game, so most of the methods of the library will take care of this. The simplest activity we can do on the screen buffer is to draw a single pixel. The steps to draw a pixel are as follows:

1. Check whether the pixel is located at the top or bottom sector of the screen buffer: for convenience, we will write a macro that returns the screen buffer sector given the line in which we want to draw.
2. Calculate the index where to write in the buffer: The buffer contains the lines of the screen one after the other, so to get the index you must multiply the line number y by the buffer width (320 pixels) and add the x coordinate.

Add this macro to GFX.h:

#define SCREENBUFFER_SECTOR(y)  (y < 256) ? 0 : 1

Also in GFX.h, add the declaration of the drawPixel method to the public section of the GFX class:

void drawPixel(uint16_t x, uint16_t y, uint8_t color);

Finally, in GFX.cpp, add the method definition:

void GFX::drawPixel(uint16_t x, uint16_t y, uint8_t color) {
    int sector = SCREENBUFFER_SECTOR(y);
    if(sector)
        y = y - 256;
    int index = 320 * y + x;
    screenBuffer[sector][index] = color;
}

Screen update

Once the screen buffer is ready, it can be transferred to the video memory to be displayed. The update method will take care of this, performing these operations for every line:

1. for each pixel translate the color index into the actual color and save it in a temporary buffer for the line;
2. send the line data over SPI, using the procedure described at the beginning of this post.

First, we need a method that sets the address window. This method uses three new commands, that we will add to the GFX.h file:

#define HX8357D_CMD_CASET       0x2A
#define HX8357D_CMD_PASET       0x2B
#define HX8357D_CMD_RAMWR       0x2C

Now we add the declaration of the setAddressWindow to the private section of the class GFX:

void setAddressWindow(uint16_t x, uint16_t y, uint16_t width, uint16_t height);

and the definition in GFX.cpp:

void GFX::setAddressWindow(uint16_t x, uint16_t y, uint16_t width, uint16_t height) {
    uint32_t columnAddress = ((uint32_t)x << 16) | (x + width - 1);
    uint32_t rowAddress = ((uint32_t)y << 16) | (y + height - 1);

    // Column address set
    digitalWrite(GPIO_HX8357D_DC, LOW);
    SPI.write(HX8357D_CMD_CASET);
    digitalWrite(GPIO_HX8357D_DC, HIGH);
    SPI.write32(columnAddress);

    // Row address set
    digitalWrite(GPIO_HX8357D_DC, LOW);
    SPI.write(HX8357D_CMD_PASET);
    digitalWrite(GPIO_HX8357D_DC, HIGH);
    SPI.write32(rowAddress);

    // Write to RAM
    digitalWrite(GPIO_HX8357D_DC, LOW);
    SPI.write(HX8357D_CMD_RAMWR);
    digitalWrite(GPIO_HX8357D_DC, HIGH);
}

Finally add the declaration of the method update in the public section of the class:

void update();

and the definition in GFX.cpp:

void GFX::update() {
    uint16_t buffer[320];
    
    // Start SPI transaction
    SPI.beginTransaction(SPISettings(HX8357D_SPI_FREQUENCY, MSBFIRST, SPI_MODE0));
    digitalWrite(GPIO_HX8357D_CS, LOW);

    // Top framebuffer sector
    for(int y=0; y<256; y++) {
        int offset = 320 * y;
        for(int x=0; x<320; x++)
            buffer[x] = palette[screenBuffer[0][offset + x]];
        setAddressWindow(0, y, 320, 1);
        SPI.writePixels(buffer, 640);
    }

    // Bottom framebuffer sector
    for(int y=0; y<224; y++) {
        int offset = 320 * y;
        for(int x=0; x<320; x++)
            buffer[x] = palette[screenBuffer[1][offset + x]];
        setAddressWindow(0, y + 256, 320, 1);
        SPI.writePixels(buffer, 640);
    }

    // End SPI transaction
    digitalWrite(GPIO_HX8357D_CS, HIGH);
    SPI.endTransaction();
}

Library test

To test the library, I wrote a small program that draws a random path on the screen:

#include <Arduino.h>
#include "GFX.h"

GFX gfx;

void setup() {
    gfx.begin();
}

int x = 160;
int y = 240;
int deltaX = 1;
int deltaY = 1;
int color = random(15);

void loop() {
    gfx.drawPixel(x, y, color);

    // 10% chance of changing x and y direction
    if(random(10) == 0)
        deltaX = random(-1, 2);
    if(random(10) == 0)
        deltaY = random(-1, 2);
    
    // update x and y position
    x += deltaX;
    if(x < 0)
        x = x + 320;
    else if(x >= 320)
        x = x - 320;
    
    y += deltaY;
    if(y < 0)
        y = y + 480;
    else if(y >= 480)
        y = y - 480;
    
    // 2.5% chance to change color
    if(random(40) == 0)
        color = random(15);
    
    // update screen
    gfx.update();
}

If everything works correctly, you should see something like this:

You can find the complete code of this post here.

In the next post…

In the next post we will start to implement some methods for drawing on the screen buffer.