Building a personal web service in Go, TDD Style: Part 4 - The Logic

With tangential issues taken care of, I was finally free to begin designing the logic of my “artist tracker” service. As I wrote about in the first entry in this series, I had been manually building a list of my favorite music artists in CSV format. After doing this for a few weeks, I had quite a few records, which I then migrated into a SQLite table called Artists. With the records in storage, I started building a way to expose the Artist resource externally.

I wanted a way to get an artist record, randomly and quickly, without needing any native SQL queries - a solution that could be expressed entirely via GORM. I settled on a combination of using the table’s COUNT and the OFFSET values from the database along with Go’s built-in random functions. First, I used GORM in my repository layer to obtain the count of the table:

 
func (ar *ArtistRepository) GetCount() (uint, error) {
    gormConn := ar.IDB.Connection()

    var count int64

    result := gormConn.Model(models.Artist{}).Count(&count)

    if result.Error != nil {
        return uint(0), result.Error
    }
    return uint(count), nil
}

With the count of total artist records, I then used Go to find a random value between 1 and the count. With the random value in hand, I went back to the repository. The concept of OFFSET in SQL databases allows us to skip X number of records when searching the database - which we can then leverage with our random value to get a random record. We limit our results to just 1 artist, as shown below:

 
func (ar *ArtistRepository) GetByOffset(offset uint) (*models.Artist, error) {
    gormConn := ar.IDB.Connection()

    var artist = models.Artist{}

    result := gormConn.Limit(1).Offset(int(offset)).Find(&artist)

    if result.Error != nil {
        return nil, result.Error
    }

    return &artist, nil
}

The result is a fast and reliable method to get a random record from our artist database. When I do not know what I want to listen to, this method is a lifesaver!

While anyone can freely read records from my service, I wanted to make sure I was the only one adding entries. To ensure I had some level of security, I created an Auth Service that would store a secret string in memory, as well as provide an authentication “green light” by matching a user provided string against the secret stored in memory. It is not the most sophisticated or secure mechanism, but it provided me with a satisfactory level of protection for my purposes.

When it came to entering new artists, I designed the logic to prevent duplicates and data errors. I wanted my entry system to automatically perform cleansing operations on user submission, and I initially set out to build that logic directly into my Artist Service. However, my Artist Service already orchestrated between the Auth Service and Artist Repository, so adding additional logic into it looked like it was going to make my testing scheme too complicated. With that, I decided to build a dependency-free logic layer called the Artist Rules.

In Artist Rules, I have a function called CleanArtistName that takes a string (artist name) as an argument, and returns a string (cleaned artist name). With no dependencies to mock, testing this is a breeze, and I can easily scale up a long list of test data for every scenario I can think of. Here are a couple examples below:

 
func TestArtistCleanName(t *testing.T) {
    var testData = []struct {
        tn   string
        name string
        want string
    }{
        {tn: "leading the", name: "the rolling stones", want: "rolling stones"},
        {tn: "sly", name: "sly and the family stone", want: "sly and the family stone"},
    }
    for _, tt := range testData {
        t.Run(tt.tn, func(t *testing.T) {
            rules := ArtistRules{}
            result, err := rules.CleanArtistName(tt.name)

            assert.Equal(t, tt.want, result)
            assert.Nil(t, err)
        })
    }
}

I discovered the need for the “Sly and the Family Stone” rule when showing the data to my wife. She saw sly and family Stone show up, and thought it was odd they were missing the. In order to not duplicate artists, the server automatically removes the first instance of the string the followed by blank space found in the artist name. This ensures that an entry of the rolling stones is cleansed and stored as rolling stones. However, the logic should have been written so that a string of the and blank space is only removed if the artist name starts with it. Edge cases like that can be tricky to find, but once that test case was added to the suite, I was confident that future changes would enforce the same desired behavior.

Given that I did not choose to use a managed SQL service, I was concerned about potential data loss or corruption - a managed service has capabilities for backup and replication out-of-box. Since I did not have that luxury, I designed an Admin Service that would take care of a nightly backup of my database to Cloud Storage.

Through some research, I found a preferred solution for SQLite using the VACUUM INTO syntax. This syntax essentially defragments the live data and stores the result into a file on disk. The resultant file can be used to restore your database, and it has the benefit of being optimally compacted.

Rob Figueiredo’s cron for golang enabled me to easily set up a 2am nightly backup. Every night, I run a SQLite command (the only raw query in my codebase) to VACUUM INTO a backup file and upload it to Cloud Storage with a name including a timestamp. If there are 3 or more copies currently in the bucket, I delete the oldest one.

My frontend was more or less hacked together quickly at the end. While the design is nothing to write home about, it is very fast and uses pure vanilla JavaScript and simple HTML and CSS. Visitors can easily see many different artist names by clicking the giant button, but they cannot access the entry form without authentication (every attempt to send data to the backend is authenticated each time, as well).

Now that I could easily store and retrieve artists, with authentication and backups, I was satisfied with what I had built. It was time to put on the finishing touches.