第 3 章

[Day 3, 4] RESP2 value and reader

RESP: Redis Serialization Protocol

We need to talk the same language when we communicate with others. When we want to communicate with Redis, we need to use RESP, which is a protocol designed for Redis.

There’re 2 versions of RESP: RESP2 and RESP3. Today, we’ll implement RESP2 because it is easier than RESP3.

Note: RESP3 is most superset of RESP2, it is supported since Redis 6.0.

The Redis serialization protocol specification is available on Redis official documentation. It tells us why and how RESP designed.

RESP Value

Magic Byte

RESP uses “magic byte” to identify data types, there’re 5 data types in RESP2:

Simple String: +
Simple Error: -
Integer: :
Bulk String: $
Array: *

Let’s define constants in pkg/resp/value.go:

https://github.com/jr-dragon/olivine/blob/

// file: pkg/resp/value.go
package resp

const(
	MAGIC_SIMPLE_STRING = '+'
	MAGIC_SIMPLE_ERROR  = '-'
	MAGIC_INTEGER       = ':'
	MAGIC_BULK_STRING   = '$'
	MAGIC_ARRAY         = '*'
)

Note: packages under pkg means public accessible for other developers who use this repo as package, read golang-standards/project-layout for more information. This guide is not a go official standard, but it is useful for mid-large project organization.

`Value` interface

It made a Value{} in Build Redis from scratch:

type Value struct {
	typ   string
	str   string
	num   int
	bulk  string
	array []Value
}

It’s simple, but I prefer define a Value interface for all RESP data types:

type Value interface {
	Marshal() []byte
}

The Marshal() is inspired from json.Marshal(), the things which be written in connection as response.

Simple String

First of all, we can make a SimpleString{} struct implements Value{} interface.

type SimpleString []byte

func (v SimpleString) Marshal() []byte {
	/*
	 * Simple String Format:
	 * |---------------------|---------|----------|
	 * | MAGIC_SIMPLE_STRING | Content | SENTINEL |
	 * |---------------------|---------|----------|
	 * |                  '+'| "hello" | "\r\n"   |
	 * |------------------------------------------|
	 */
	var buf bytes.Buffer

	buf.WriteRune(MAGIC_SIMPLE_STRING)
	buf.Write(v)
	buf.WriteString(SENTINEL)

	return buf.Bytes()
}

Write to `net.Conn`

We are using conn.Write([]byte("ACK: " + strings.ToUpper(msg))) to write response in internal/server/server.go

func (s *simpleSrv) serve(conn net.Conn) {
	defer conn.Close()

	rd := bufio.NewReader(conn)
	msg, err := rd.ReadString('\n')
	if err != nil {
		slog.Error("failed to read from conn:", slog.Any("error", err))
		return
	}

	time.Sleep(time.Second * 3)

	if _, err := conn.Write([]byte("ACK: " + strings.ToUpper(msg))); err != nil {
		slog.Error("failed to write to conn:", slog.Any("error", err))
		return
	}
}

-	if _, err := conn.Write([]byte("ACK: " + strings.ToUpper(msg))); err != nil {
+	if _, err := conn.Write(resp.SimpleString("OK").Marshal()); err != nil {

func (s *simpleSrv) serve(conn net.Conn) {
	defer conn.Close()

	rd := bufio.NewReader(conn)
	_, err := rd.ReadString('\n')
	if err != nil {
		slog.Error("failed to read from conn:", slog.Any("error", err))
		return
	}

	if _, err := conn.Write(resp.SimpleString("OK").Marshal()); err != nil {
		slog.Error("failed to write to conn:", slog.Any("error", err))
		return
	}
}

Manually testing with `redis-cli`

Redis provides some useful tools, the redis-cli is the most popular tool, it can send requests and handle responses with RESP, we can check whether our service is compatible with Redis or not.

$ redis-cli -p 16879 ping
OK

Redis Command: Array and BulkString

There’s an RESP array with bulk strings in requests, it calls “Redis Command”.

Array{ BulkString{"PING"} }: a PING command.
Array{ BulkString{"GET"}, BulkString{"bar"} }: a GET command with "bar" argument.

`BulkString{}` Value

There’re 2 types of bulk string in RESP:

Because Null type is defined in RESP3, Redis uses “Null Bulk String” type for null string value.

// file: pkg/resp/value.go
const (
	MAGIC_SIMPLE_STRING = '+'
	MAGIC_SIMPLE_ERROR  = '-'
	MAGIC_INTEGER       = ':'
	MAGIC_BULK_STRING   = '$'
	MAGIC_ARRAY         = '*'
+	SENTINEL            = "\r\n"
)

type BulkString struct {
	null bool
	data []byte
}

func (v BulkString) Marshal() []byte {
	/*
	 * Null Bulk String Format:
	 * |-------------------|--------|----------|
	 * | MAGIC_BULK_STRING | LENGTH | SENTINEL |
	 * |-------------------|--------|----------|
	 * |               '-' | "-1"   | "\r\n"   |
	 * |-------------------|--------|----------|
	 */
	if v.null {
		return []byte("$-1\r\n")
	}

	/*
	 * Non Null Bulk String Format:
	 * |-------------------|--------|---------|----------|
	 * | MAGIC_BULK_STRING | Length | Content | SENTINEL |
	 * |-------------------|--------|---------|----------|
	 * |               '$' |    "5" | "hello" | "\r\n"   |
	 * |-------------------|--------|---------|----------|
	 */
	var buf bytes.Buffer
	buf.WriteRune(MAGIC_BULK_STRING)
	buf.WriteString(strconv.Itoa(len(v.data)))
	buf.WriteString(SENTINEL)
	buf.Write(v.data)
	buf.WriteString(SENTINEL)
	return buf.Bytes()
}

`Array{}` Value

Like BulkString{}, RESP2 defeined null array.

type Array struct {
	null bool
	data []Value
}

func (v Array) Marshal() []byte {
	/*
	 * Null Array Format:
	 * |-------------|--------|----------|
	 * | MAGIC_ARRAY | LENGTH | SENTINEL |
	 * |-------------|--------|----------|
	 * |         '*' | "-1"   | "\r\n"   |
	 * |-------------|--------|----------|
	 */
	if v.null {
		return []byte("*-1\r\n")
	}

	/*
	 * Array Format:
	 * |-------------|--------|----------|-----------------------|
	 * | MAGIC_ARRAY | LENGTH | SENTINEL | Marshaled Element [0] |
	 * |-------------|--------|----------|-----------------------|
	 * |         '*' |    "1" | "\r\n"   | "$4\r\nPING\r\n"      |
	 * |-------------|--------|----------|-----------------------|
	 */
	var buf bytes.Buffer
	buf.WriteRune(MAGIC_ARRAY)
	buf.WriteString(strconv.Itoa(len(v.data)))
	buf.WriteString(SENTINEL)
	for _, v := range v.data {
		buf.Write(v.Marshal())
	}
	return buf.Bytes()
}

Read from `net.Conn`

We use *bufio.Reader to handle connections at this moment, but it is not designed for RESP.

Consider the reader below:

rd := bufio.NewReader(conn)
for {
	msg, _, err := rd.ReadLine()
	if err != nil {
		fmt.Printf("err = %+v\n", err)
		break
	} else {
		fmt.Printf("msg = %s\n", msg)
	}
}

When I use *1\r\n$4\r\nPING\r\n as Redis Command, I needs to parse msg multiple times — NOT ELEGANT.

Demo: Go Playground

Therefore, I made *resp.Reader in pkg/resp/io.go to solve this problem.

type Reader struct {
	rd *bufio.Reader
}

func NewReader(rd io.Reader) *Reader {
	return &Reader{rd: bufio.NewReader(rd)}
}

func (r *Reader) Read() (Value, error) {
	t, err := r.rd.ReadByte()
	if err != nil {
		return nil, err
	}

	switch t {
	case MAGIC_ARRAY:
		return r.readArray()
	case MAGIC_BULK_STRING:
		return r.readBulkString()
	default:
		return nil, fmt.Errorf("%w: %c", ErrUnknownType, t)
	}
}

`readLine()` and `readInt()`

As we know, the “Redis Command” is an “Array of BulkStrings” (it calls “multibulk” in redis internal), and both “Array” and “BulkString” formats by {MAGIC}{LENGTH}\r\n{DATA}\r\n.

It nice to have readLine() and readInt() before we implement readArray() and readBulkString().

func (r *Reader) readLine() ([]byte, error) {
	var buf bytes.Buffer

	for {
		data, err := r.rd.ReadBytes('\n')
		if err != nil {
			return nil, err
		}

		buf.Write(data)
		if data[len(data)-2] == '\r' {
			return buf.Bytes()[:buf.Len()-2], nil
		}
	}
}

func (r *Reader) readInt() (int, error) {
	line, err := r.readLine()
	if err != nil {
		return 0, err
	}

	return strconv.Atoi(string(line))
}

`readBulkString()`

BulkString is designed as a binary safe string, reading a binary safe string depends on its length defeined in header.

Note: C-style string is a char array and terminates with null byte ‘\0’, it is not binary-safe, because given data could includes null byte anywhere.

func (r *Reader) readBulkString() (BulkString, error) {
	sz, err := r.readInt()
	if err != nil {
		return BulkString{}, err
	}
	if sz < 0 { // Ref: https://redis.io/docs/latest/develop/reference/protocol-spec/#null-bulk-strings
		return BulkString{null: true}, nil
	}

	buf := make([]byte, sz)
	if _, err := io.ReadFull(r.rd, buf); err != nil {
		return BulkString{}, err
	}

	// Drop tailing "\r\n" from reader.
	r.rd.ReadByte()
	r.rd.ReadByte()

	return BulkString{data: buf}, nil
}

`readArray()`

RESP array can be imaged as “collection of RESP values”, we can read recursively for each data in collection.

func (r *Reader) readArray() (Array, error) {
	sz, err := r.readInt()
	if err != nil {
		return Array{}, err
	}
	if sz < 0 { // Ref: https://redis.io/docs/latest/develop/reference/protocol-spec/#null-arrays
		return Array{null: true}, nil
	}

	arr := Array{}
	arr.data = make([]Value, 0, sz)

	for range sz {
		v, err := r.Read()
		if err != nil {
			return arr, err
		}

		arr.data = append(arr.data, v)
	}

	return arr, nil
}

Handle Redis Commands

We are using msg, err := rd.ReadString('\n') in internal/server/server.go, it’s time to use *resp.Reader instead.

	rd := resp.NewReader(conn)
-	msg, err := rd.ReadBytes('\n')
+	cmd, err := rd.Read()

func (s *simpleSrv) serve(conn net.Conn) {
	defer conn.Close()

	rd := resp.NewReader(conn)
	cmd, err := rd.Read()
	if err != nil {
		slog.Error("failed to read from conn:", slog.Any("error", err))
		return
	}

	slog.Info("Read RESP command:", slog.Any("command", cmd))

	if _, err := conn.Write(resp.SimpleString("OK").Marshal()); err != nil {
		slog.Error("failed to write to conn:", slog.Any("error", err))
		return
	}
}

RESP: Redis Serialization Protocol

RESP Value

Magic Byte

Value interface

Simple String

Write to net.Conn

Manually testing with redis-cli

Redis Command: Array and BulkString

BulkString{} Value

Array{} Value

Read from net.Conn

readLine() and readInt()

readBulkString()

readArray()