fancy thoughts

In this post, I'm going to show you how to install mpsyt on a Debian machine. I haven't been this happy about an app in a long time. I spend a lot of time listening to mixes on youtube, and have always been bothered by how much CPU and RAM Firefox requires to do so. With this tool I can easily search and listen to audio from youtube while using very little RAM and CPU cycles.

install

mpsyt has a few dependencies it uses to make things work. We start by installing our debian packages these are pip3 and mplayer. pip3 being the python3 package manager, and mplayer being a comandline audio playback tool.

sudo apt install python3-pip mplayer

Once we have pip3 installed we use it to install mps and youtube-dl. youtube-dl has gone through a lot of changes since the release of Buster. pip3 will have the most up to date package.

pip3 install mps-youtube youtube-dl

Then open up mpsyt in the terminal. You can enter a search by starting the query with '/'.

/com truise fairlight

You can choose which of the results you want by entering in a number into the prompt.

Then you have it, a youtube audio streamer in the terminal that doesn't use up all of your computer's resources.

Continuing my exploration of suckless tools I wanted to try compiling a program with tcc and musl. tcc stands for Tiny C Compiler, and musl is a lightweight libc alternative to glibc. I was generally sold on the principal of these tools, but I could not for the life of me find documentation on how to get these two working together.

Turns out it is rather quite trivial if you use the right OS. In my case, I was doing this on Void Linux running inside of a QEMU vm. The reason I chose Void, was because it has a musl build, so all of the tools are built with musl instead of glibc. musl-devel is not available on the glibc version.

On Void Linux, to install the necessary dependencies:

sudo xbps-install -S tcc musl-devel

What tripped me up is that on Void's packages directory page, I had not selected x86_64-musl so I couldn't find musl-devel. Once you get these two packages installed you are in business.

Write your hello world like this:

#include <stdio.h>

int main() {
  printf("Hello, World!\n");
  return 0;
}

Then with tcc we can treat this like a script and execute it:

tcc -run hello.c

That is it! Hopefully, this saves some time for other noobs like me.

Lately, I've had a new appreciation for minimalist C applications. There is a certain Zen quality about C that I really like. Knowledge of the possible “footguns” actually makes me more focused and insistent on writing good code. It is a bit like weight training but for programmers. Coding in other languages like Python feels trivial after spending a bit of time wrangling some C. less is one of the original Unix terminal applications for viewing files. It is a good example of the Unix philosophy: “Do one thing, and do it well”. It differentiates itself from editors by not loading the entire file into ram. The wikipedia page has a very detailed explanation. If you haven't used less before, I would recommend giving it a spin before continuing with the rest of this post.

In summary what we will be making is a small terminal application that uses a file stream to view text. If you want to follow along check out the final repo here on Sourcehut. We will be using termbox for rendering and general terminal application behavior. termbox is an alternative to the popular curses library and is recommend by suckless. It is very small and easy to use.

On Debian we can install it like so:

sudo apt install libtermbox-dev

Once we have that installed we will start with a simple hello world using termbox. For compilation I am using a very basic Makefile that calls out to my default c compiler. On my machine it is gcc. I'm using c99 because people smarter than me have come to a consensus that it is best. Also notice that in the build command I am linking the termbox library. I decided to name this application bless because it is “basically less” :p.

all: build

build:
	$(CC) -o bless bless.c -Wall -W -pedantic -std=c99 -ltermbox

clean:
	rm bless

Now the actual hello world looks like this. Most of this code is basic boiler plate to get termbox up and running. We initialize termbox, render our text, and then wait for the escape key to shutdown. Easy!

#include <termbox.h>
#include <stdio.h>

void render_line(char *text) {
	int i = 0;
	while (text[i] != '\0') {
		tb_change_cell(i, 0, text[i], TB_WHITE, TB_DEFAULT);
		i++;
	}
}

int main() {

	int ret = tb_init();
	if (ret) {
		fprintf(stderr, "tb_init failed with error code %d\n", ret);
		return 1;
	}

	tb_clear();
	tb_select_output_mode(TB_OUTPUT_NORMAL);

	render_line("Hello, World!\n\0");

	tb_present();

	struct tb_event ev;
	while (tb_poll_event(&ev)) {
		switch (ev.type) {
		case TB_EVENT_KEY:
			switch (ev.key) {
			case TB_KEY_ESC:
				goto done;
				break;
			}
			break;
		}
	}

done:
	tb_shutdown();

	return 0;
}

Now we are going to step our game up by rendering a file in the limits of our terminal screen. The filename is given as user input. So lets change our main function to accept a filename and attempt to open it.

int main(int argc, char **argv) {
	(void)argc; (void)argv;
	FILE *fp = fopen(argv[1], "r");

When we shutdown termbox we also want to close the file:

done:
	tb_shutdown();
	fclose(fp);

The next step is to add the capability of rendering the file. We do this by replacing renderline with renderfile, which will accept a file pointer instead a character string.

void render_file(FILE *fp) {
	int line_count = 0;
	int x = 0;
	while (line_count < tb_height()) {
		char c = fgetc(fp);
		if (c != EOF) {
			if (c == '\n' || x == tb_width()) {
				line_count++;
				x = 0;
			} else {
				if (c == '\t') {
					x+=8;
				} else {
					tb_change_cell(x, line_count, c, TB_WHITE, TB_DEFAULT);
					x++;
				}
			}
		} else {
			break;
		}
	}
}

Since all we need to do is render a portion of the file stream we can simply loop until we reach the terminal height (tb_height) or reach the end of the file. In order to do basic line wrapping we check if our x coordinate is equivalent to the terminal width (tb_width). Another thing to note is we have to manually handle tabs or else termbox will have rendering bugs. In this case we simply skip over 8 spaces.

Now when we execute bless we pass in the name of the file we want to view.

./bless *filename*

Great we can view the file in our terminal. However, you will quickly notice we can't view parts of the file that are greater than the terminal height. Lets add some navigation features to fix this.

We will start by adding scroll down by pressing the down arrow key. To do this, we will need to add an event handler for when we press the down arrow.

case TB_KEY_ARROW_DOWN:
	cursor = scroll_down(fp, cursor);
	render(fp, cursor);
	break;

You'll notice that we are calling two new functions when the down arrow is pressed. scroll_down will set the file stream at the beginning of the next line, and return an integer. That integer is a simple variable called cursor which tracks our position in the file. Once we change position we have to render the new view.

I'll start by breaking down render. It mostly includes termbox boiler plate and we call the render_file inside of it. The reason we need this is that when we scroll the characters need to be reset or else we will see artifacts from the last scene.

void render(FILE *fp, int cursor) {
	tb_clear();
	tb_select_output_mode(TB_OUTPUT_NORMAL);
	render_file(fp);
	tb_present();
	fseek(fp, cursor, SEEK_SET);
}

After rendering the file we set the file position to the cursor offset since it changes every time we call fgetc.

The scroll down function looks like this:

int scroll_down(FILE *fp, int cursor) {
	char c;
	while (c != '\n') {
		c = fgetc(fp);
		cursor++;
	}
	fseek(fp, cursor, SEEK_SET);
	return cursor;
}

scroll_down loops from the cursor position until it finds a newline, and sets the file position at the new cursor position. Which will be at the start of the next line.

How about scrolling up? Well that is a little more tricky since we fgetc only goes down the file stream, not up. Using a combination of fseek and fgetc we can implement scrolling up in a concise manner.

The implementation of scroll up looks like this:

int scroll_up(FILE *fp, int cursor) {
	char c;
	while (c != '\n' && cursor != 0) {
		cursor--;
		fseek(fp, cursor, SEEK_SET);
		c = fgetc(fp);
	}
	return cursor;
}

And voila! Basically less with scroll down and up features in less than 100 lines of code. After messing around with this you may notice a few bugs, but for simplicity sake I think this implementation is okay. The real version of less has a lot more features, so you probably won't want to use this for personal use. I hope this post has inspired to make your own termbox app in C. At the very least it was a good learning experience for myself.

Recently I have been interested in user friendly, self hosting solutions. After a bit of web surfing, I found this was a much larger space than I had anticipated. It seems that in the last decade a variety of projects have been popping up, and with increasing frequency. Some of the buzzwords I came across where “personal cloud”, and “PaaS” or “Platform as a Service”. It is always fun to poke fun at buzzwords, but I think these terms are relevant when attempting to portray these ideas to a larger audience. This is still very niche and weird. Quite literally the best resource for a list of self hosting solutions was this kind of creepy youtube video.

The basic idea is to be able to easily deploy web services on a common platform. I'm partial to Debian based solutions, so Yunohost and FreedomBox stood out to me. This post is dedicated to Yunohost.

Overall, I like Yunohost a lot. It is fun, and easy to use. I spent a whole day installing and uninstalling apps to try things out.

What is Yunohost?

The Yunohost FAQ states:

It provides a software that aim to make it easy for people to run and administrate their own server, with minimal knowledge and time required.
YunoHost may be called a distribution or an operating system, but it's actually "just" a simple layer added over the top of Debian, which does most of the hard work for you.

FYI: The name Yunohost is pronounced, “Why You No Host?”.

Installation

The insallation instructions I followed for Debian Stretch can be found here.

I chose to run my installation on a brand new Debian cloud image with only 2G of RAM. For initial demo purposes, I noticed no serious performance issues on this machine. The first thing that got stuck me was that I was not able to install on Buster. There documentation only says Stretch, so I should've known better. You can track the status of this issue here. Besides this hiccup, the installation process was mostly a breeze. The documentation for this is very detailed. The docs are a wiki, so if you notice an issue you can actually submit patches directly from the site. I submitted a few patches myself.

To install on a Debian Stretch machine, simply run:

curl https://install.yunohost.org | bash

If you are concerned with this approach, they have a disclaimer on the docs about this. Essentially, what you are downloading is a Debian metapackage that will install the necessary dependencies. It is quite opinionated on how it does the configuration, which is why it is recommended to run this on a dedicated machine.In my opinion, having the yunohost app be an official Debian package would alleviate a lot of these concerns. It would be more intuitive for Debian users, especially when they advertise as being Debian based. I would like to be able to install the yunohost app with apt.

Out of the Box

Upon installation, Yunohost comes with a Single Sign On service, email server, xmpp server, and the admin GUI used to deploy applications.

When originally accessing the web GUI your browser will complain about the certificate. Find a way to bypass this and then you can login. Once there, you can go to Domains and set up a Let's Encrypt certificate for your domain.

The basic setup allowed for me to connect Thunderbird to the email server and send messages. These messages were sent to the Spam folders of the accounts I tested it with. With more domain hacking I should be able to fix this.

The Single Sign On Service, called SSOwat, is one of my favorite features of Yunohost. I haven't seen anything similar advertised by other projects. The implementation is a NGINX plugin written in Lua. Fine grained application access is controlled by SSOwat. Only users who are registered with the Yunohost service will be able to access these sites if the application is not made public. Easy setup private network is a great feature.

Applications

I tried a handful of applications. The official apps actually work out of the box. Others work with varying degrees of success and they try to be very transparent about this.

One thing that is confusing, is it isn't always clear how the domains should be setup for each app. Some apps are okay with a url route like my.domain.org/riot, others require a subdomain like matrix.domain.org. The yunohost app does catch these mistakes if you mess up It would be nice if this situation was more explicit though.

Now I'll go through a list of apps that I tried.

Matrix/Riot

matrix-synapse and Riot are both easy to install with basic features working. The Matrix Synapse server that is installed does not come with a GUI (that is what Riot is for) you can access it with other Matrix clients like Fractal or Riot. I wanted to give the full experience a try, so I installed Riot along with it. This was quite easy to do. You can optionally put the Riot app behind a private network as well. I will probably be using this very soon for a small group of friends.

Pleroma

My experience with manually installing Pleroma a year ago was painful, so I had high hopes for this. Those expectations were met. The package for this installation was successful and the app seemed to work out of the box. Very pleased to see this only took a couple clicks to install.

Personally, I prefer Pleroma over Mastodon for a variety of reasons. Most of them being related to the implementation differences (Phoenix vs Rails). I'm a big fan of Elixir and like that Pleroma uses less resources than Mastodon. There are also no character limits... Using less resources on something like Yunohost is crucial. These two technologies are a good fit imo.

WriteFreely

WriteFreely is a simple blogging service (the one I use for this blog). It is written in Golang with a minimalist and clean design. Unfortunately the package did not install successfully so I couldn't try this out. When this gets resolved I might migrate this blog to Yunohost.

Mobilizon

Mobilizon is a Pleroma fork for coordinating events. This is something I need, and a lot of other people need. The app was able to install, however, the email Registration did not work. This is likely a fault of my own, however I had no issue with Riot. Not sure what deal is with that.

Custom Web App

I have not tried this feature out, but I wanted to point it out because I think it is neat. The real value Yunohost provides is not having to deal with server setup and certificates. So having a tool like Yunohost to host multiple sites with is really cool. Yunohost is really helpful for a hobbyist application developer trying to get things started. It is also possible to configure a simple redirect for a domain name to a particular url, even localhost. Could be fun to write a post detailing how to do this. This is where Yunohost starts venturing into Platform as a Service with these kinds of features.

Conclusion

Yunohost does a great job at lowering the barrier to entry for deploying all of these great open source applications. Since the domain stuff is still a bit tricky, I think that the target user is still someone with a technical background. By no means do they have to be a developer, just someone who isn't too afraid of Linux, the terminal, and DNS configs. If you are tired of annoying deployment setups then I recommend trying this out. There are a lot more bits and pieces I didn't cover here, like the intuitive UI and great branding. Try it yourself, you wont regret it.

Additional Sources

This post will layout the steps for setting up an internet radio using only Debian packages. The last part of it will go over how to create a Golang proxy server to make a fun website to host your stream.

One of my other hobbies besides programming is music. I love listening to music, making music, and even curating music. One thing I've always wanted to participate in is a silent disco, either as a listener or a DJ. If you aren't familiar with a silent disco, it is simply a group of people all listening to the same music via headphones. So I thought to myself how hard might it be to make one myself? Turns out either really hard or fairly easy depending on the approach you take.

My original idea was to basically stream my laptop's audio input to a browser via a web server using Golang. What I've learned is this is actually quite difficult to build from scratch. Thus resulting in me taking a different, but much easier approach. What I ended up doing was use already available Debian packages to do all of the heavylifting. The two packages I used were icecast2 and darkice.

Icecast

Icecast is the streaming server used to send streams to listeners. It is actually really cool, has a lot of features, and has been around for a couple of decades. The neat thing about is you can host the icecast server somewhere, and use a source client to stream audio to it from anywhere in the world. Icecast + source client (darkice) = internet radio. It is really that easy. On top of that, it has a dead simple installation process on Debian.

sudo apt install icecast2

When installing, a prompt comes up to set config params (mostly passwords). For experimenting purposes I used the default settings. However, you should certainly make the passwords more secure if you are wanting to put this in production. If you need to find the config file again, it is located at /etc/icecast2/icecast.xml. You will need root access to edit it. Once the installation is complete, the icecast server will run in the background. Now all you need is a source client to stream audio to it. That is where darkice comes.

Darkice

darkice is the software used to stream audio to icecast, also known as a source client. There are a lot of other source clients out there. And I can't remember exactly why I chose darkice (might've been the cool name), but it did seem to be somewhat popular, so I was able to find links for blogs and docs to get me started. darkice is also very easy to install on Debian:

sudo apt install darkice

For testing things out, I simply installed everything on one machine. The main thing to keep in mind is that darkice is the client, and icecast is the server. darkice has a little bit more of an involved configuration, and the official website is not super helpful in this regard. Luckily, I did find this post to be very helpful in getting my configuration going. The default behavior of darkice on Debian is to look for the config file at /etc/darkice.cfg. Upon installation, this file does not exist. So you have to write it yourself. Once again make sure you have root access.

sudo nano /etc/darkice.cfg

Here is what my config file looks like:

[general]
duration      = 0       # duration of encoding, in seconds. 0 means forever
bufferSecs    = 2       # size of internal slip buffer, in seconds
reconnect     = yes     # reconnect to the server(s) if disconnected
 
# this section describes the audio input that will be streamed
[input]
device          = default # OSS DSP soundcard device for the audio input
sampleRate      = 44100   # sample rate in Hz. try 11025, 22050 or 44100
bitsPerSample   = 16      # bits per sample. try 16
channel         = 2       # channels. 1 = mono, 2 = stereo

# this section describes a streaming connection to an IceCast2 server
# there may be up to 8 of these sections, named [icecast2-0] ... [icecast2-7]
# these can be mixed with [icecast-x] and [shoutcast-x] sections
[icecast2-0]
bitrateMode     = cbr     # constant bit rate, or else vorbis doesn't work well with Android
format          = vorbis     # format of the stream: ogg vorbis
bitrate         = 128     # bitrate of the stream sent to the server
quality         = 0.4
server          = localhost # server url
         
# host name of the server
port            = 8000      # port of the IceCast2 server, usually 8000
password        = hackme # source password to the IceCast2 server
mountPoint      = stream  # mount point of this stream on the IceCast2 server, 'stream' is default
name            = Fancy Cade Radio  # name of the stream

description     = Cool radio # description of the stream
url             = http://www.myradio.net # URL related to the stream
genre           = various               # genre of the stream
public          = yes                   # advertise this

Once that is saved. All you have to do is run darkice in the terminal:

darkice

If you navigate to localhost:8000/stream you will be able to listen to your stream. If you have other devices on the same wifi they can connect as well. All you have to do is replace localhost with your machine's hostname (sometimes only the ip address works). I chose to use ogg vorbis for the audio stream as opposed to mp3 because that is how I roll, Free Software all the way!

To get hostname:

hostname

To get ip address:

hostname -I

Grab some headphones and you have a silent disco in the comfort of your own home!

Golang Proxy Server

If you want to just use the front end provided by icecast then you can ignore the rest of this post. However, if you want to use the stream provided by icecast as a resource in your own website then this next part will show you how. I chose to use Golang because it is exceptionally good at making simple, lightweight web services with no external dependencies.

You can see an example of a finished product here. The code is hosted on Sourcehut (if you haven't checked out Sourcehut, you totally should).

The key ingredient is piping the icecast stream to a route on your server. Which is done like this:

func handleAudioStream(w http.ResponseWriter, r *http.Request) {                                                                                               
        read, write := io.Pipe()                                                                                                                               
                                                                                                                                                               
        go func() {                                                                                                                                            
                defer write.Close()                                                                                                                            
                resp, err := http.Get("http://localhost:8000/stream"); if err != nil {                                                                         
                        return                                                                                                                                 
                }                                                                                                                                              
                defer resp.Body.Close()                                                                                                                        
                io.Copy(write, resp.Body)                                                                                                                      
        }()                                                                                                                                                    
                                                                                                                                                               
        io.Copy(w, read)                                                                                                                                       
} 

We request the audio stream from the icecast server, and pipe the output to the ResponseWriter.

The route looks like this:

http.HandleFunc("/audio", handleAudioStream)

On the front end we can use the audio html element to play the stream.

<audio id="audio">                                                                                                                                         
  <source src="/audio" type="audio/ogg">                                                                                                                   
</audio>

Note: This will work on everything besides iOS devices. You can thank Apple for not following modern web standards.

Nginx Forwarding

When you want to take your silent disco to the global stage. You will have to do a couple of extra steps. Specifically, you will need to run icecast2 and the silentdisco app on a server. I chose to use nginx+lets encrypt to handle HTTPS requests, because it is battle tested and well documented. The important piece in this step is that darkice needs access to icecast2. However, if you setup the reverse proxy to only redirect to silentdisco then it won't work. My solution was to simply add a route to my nginx config, that would redirect the stream request to icecast instead of the silentdisco golang app. It Add these lines to your nginx server config:

location /stream {
    proxy_pass  http://127.0.0.1:8000;
}

Restart your nginx server:

sudo systemctl restart nginx

Now listeners from around the world can listen to sweet tracks!

Conclusion

I originally had the idea of making an audio visualizer. However, it ended up making the front end code overly complicated and a bit buggy. Specifically, the MediaElementSource would fail silently for whatever reason, and take the audio stream down with it. Not cool, man! Wanted to point this out, in case anyone had a similar idea. Overall, I'm happy with my minimalist design. It loads fast and is light on resources. Which is a bigger concern for mobile listeners. We don't want the party to stop just because of low batteries!

So there you have it. A DIY internet radio with free software, Debian, and a little bit of Golang. Hopefully, this post enables you to have some good times with your friends!

This is fancycade, signing off.

In this post I wanted to go over a few CLI utilities written in Rust that I am excited about. There are a lot of different Rust CLI tools out there, but I wanted to focus on ones that replace some of the GNU Core Utilities with Rust alternatives. The tools I will be talking about are exa, ripgrep, and bat.

You may be wondering why would you need alternatives for trusted utilities that have been around for decades? Well, a few reasons. First, they are written in Rust! Which for me is reason alone to use them. Secondly, some of these tools are makeovers with pretty colors and sane defaults, and sometimes they are even faster than the original. Some of the effort behind these tools goes to support the RedoxOS project which I highly recommend checking out if you haven't.

Side note: If you want to follow along I'm assuming you are using the Debian operating system. You can also install these tools with cargo directly.

Side note 2: Unfortunately, I wasn't able to show the proper colors for these tools, because of the limitations of my blogging skills. Just keep in mind that each of these tools has great color schemes that significantly help readability.

exa

exa is a cli tool for listing files in a directory. It serves to be an alternative to the ls command. The improvements I would consider are nice colors, sane defaults, and extra goodies.

It can be installed like so:

sudo apt install exa

Besides using ls to list files you can also see how big a file is. At my job I work with a lot of large csv and json files, so this comes in handy. The only downside to this is that to list the filesizes as not bytes you have to do this incantation:

fancycade[test_dir]: ls -l --block-size=M
total 1M
-rw-r--r-- 1 shiba shiba 1M Sep  1 14:25 data.csv
-rw-r--r-- 1 fancycade fancycade 0M Sep  1 14:03 README
drwxr-xr-x 2 fancycade fancycade 1M Sep  1 14:03 src

Which only gives the file size in Megabytes. Since it rounds down or up it doesn't giving clear information as to the actual size of your file.

Whereas exa chooses the appropriate unit based on the file size without listing the unit, while also being much easier to remember.

fancycade[test_dir]: exa -l
.rw-r--r-- 786k fancycade  1 Sep 14:25 data.csv
.rw-r--r--  182 fancycade  1 Sep 14:24 README
drwxr-xr-x    - fancycade  1 Sep 14:03 src

Another feature that exa has that ls doesn't have is the ability to list the files in a directory as a tree. Normally this would be done with the tree utility, but now we don't even need to install it if we have exa. For example:

fancycade[test_dir]: exa --tree
.
├── data.csv
├── README
└── src
   └── generate_data.rb

Typical tree representation coming from the same tool. exa is pretty sweet, highly recommend giving it a try.

ripgrep (rg)

ripgrep is a Rust alternative to the well known tool grep. Who could've guessed?! Now this tool is one that I'm the least familiar with because I'm not a master grep user. However, I have found a few use cases that I find to be very helpful. The cool thing about ripgrep is that it is actually faster than the original. Which is crazy, since grep was always touted as extremely fast.

To install ripgrep you simply do this:

sudo apt install ripgrep

Now to use ripgrep we type the alias rg. Imagine a scenario where you are looking for a file in a directory that has a lot of stuff in it. You run exa and all you see is a wall of text. One clever trick is to pipe the output of exa into ripgrep to filter only the relevant filenames. So if I know I'm looking for a csv file, I can do something like this:

fancycade[test_dir]: exa | rg csv
data.csv

Instead of printing out all the contents of the current directory it only outputs files that include csv in the filename. This is only the tip of the iceberg of what you can do with this tool, so I highly suggest experimenting and trying things out.

bat

bat is a Rust alternative to the cat utility with a few extra goodies. It also combines the features of less as well, so it can be considered a cross between the two. The problem with cat is that it can be hard to read large files. Which is why less is useful. On top of that bat provides formatting for a lot of common filetypes like markdown or source code. And line numbers! Which is really helpful when looking at tabular data such as csvs.

Unfortunately, as of this post, bat isn't in Debian stable (buster). However, there is a workaround, and it will probably be available in Ubuntu fairly soon. This workaround is not typically recommended by the Debian community because it does not respect their ideas of a “stable” version. So make sure to do this with caution.

We begin the process by editing our /etc/apt/sources.list with nano (make sure you have proper permissions):

sudo nano /etc/apt/sources.list

Add this line to the file:

deb http://deb.debian.org/debian/ sid main

This allows apt to get packages in the Debian unstable (aka sid) archive.

You have to update the package lists:

sudo apt update

Now you can install bat:

sudo apt install bat

As I mentioned earlier, it is not recommended to use packages from Debian unstable so I highly suggest removing the line we added to the /etc/apt/sources.list because we don't want to accidentally grab other unstable packages later.

Once we have bat installed we can get started. We can look at a csv file (list of popular baby names) like this:

fancycade[test_dir]: bat data.csv
───────┬──────────────────────────────────────────────────────
       │ File: data.csv
───────┼──────────────────────────────────────────────────────
   1   │ Year of Birth,Gender,Ethnicity,First Name,Count,Rank
   2   │ 2016,FEMALE,ASIAN AND PACIFIC ISLANDER,Olivia,172,1
   3   │ 2016,FEMALE,ASIAN AND PACIFIC ISLANDER,Chloe,112,2
   4   │ 2016,FEMALE,ASIAN AND PACIFIC ISLANDER,Sophia,104,3
   5   │ 2016,FEMALE,ASIAN AND PACIFIC ISLANDER,Emma,99,4
   6   │ 2016,FEMALE,ASIAN AND PACIFIC ISLANDER,Emily,99,4
   7   │ 2016,FEMALE,ASIAN AND PACIFIC ISLANDER,Mia,79,5
   8   │ 2016,FEMALE,ASIAN AND PACIFIC ISLANDER,Charlotte,59,6
   9   │ 2016,FEMALE,ASIAN AND PACIFIC ISLANDER,Sarah,57,7
  10   │ 2016,FEMALE,ASIAN AND PACIFIC ISLANDER,Isabella,56,8
  11   │ 2016,FEMALE,ASIAN AND PACIFIC ISLANDER,Hannah,56,8
  12   │ 2016,FEMALE,ASIAN AND PACIFIC ISLANDER,Grace,54,9
  13   │ 2016,FEMALE,ASIAN AND PACIFIC ISLANDER,Angela,54,9
  14   │ 2016,FEMALE,ASIAN AND PACIFIC ISLANDER,Ava,53,10
  15   │ 2016,FEMALE,ASIAN AND PACIFIC ISLANDER,Joanna,49,11
  16   │ 2016,FEMALE,ASIAN AND PACIFIC ISLANDER,Amelia,44,12
  17   │ 2016,FEMALE,ASIAN AND PACIFIC ISLANDER,Evelyn,42,13
  18   │ 2016,FEMALE,ASIAN AND PACIFIC ISLANDER,Ella,42,13
  19   │ 2016,FEMALE,ASIAN AND PACIFIC ISLANDER,Arya,42,13
  20   │ 2016,FEMALE,ASIAN AND PACIFIC ISLANDER,Ariana,40,14
  21   │ 2016,FEMALE,ASIAN AND PACIFIC ISLANDER,Maya,39,15
  22   │ 2016,FEMALE,ASIAN AND PACIFIC ISLANDER,Alina,39,15
  23   │ 2016,FEMALE,ASIAN AND PACIFIC ISLANDER,Fiona,35,16
  24   │ 2016,FEMALE,ASIAN AND PACIFIC ISLANDER,Ashley,34,17
  25   │ 2016,FEMALE,ASIAN AND PACIFIC ISLANDER,Anaya,34,17
  26   │ 2016,FEMALE,ASIAN AND PACIFIC ISLANDER,Fatima,34,17
:

Notice the line numbers on the left. Like less we can navigate the file with up/down arrows and exit by pressing 'q'.

bat is really great, and has become something I use on a daily basis. It is amazing what subtle differences in tooling can do to your workflow. Hopefully, you found this post interesting, and are excited to try cli tools written in Rust like I am.

This is fancycade, signing off.

Since this is my first post I thought it would only make sense to write a tutorial on how to setup the WriteFreely web app which is what I'm using to host this blog. IMO, it is best to write documentation while it is still fresh in one's mind.

One goal of mine for this blog is to spread information on free software that empowers individuals. To me, WriteFreely is a great example of this. To be specific WriteFreely is a federated web application written in Go for hosting and writing blogs. The federated part is cool because links will integrate with other federated applications like Mastodon or Pleroma. Go is cool because it is light in resource usage so it is possible to run the app on a small server. This site is run on a 2G Digital Ocean droplet running Debian. I will be talking about how great Digital Ocean is throughout this post, but just to be clear, I am in no way affiliated, simply a satisfied customer.

Debian is the obvious choice for me, because it is my favorite Linux distro. Debian is one of the most successful free software projects ever, and it has a lot of history. The nice thing about Digital Ocean is they allow a variety of OS's to choose from, and they already have support for Debian 10 (buster) including great documentation that I used myself to get this site going.

WriteFreely is pretty neat, and their getting started guide is fairly useful, but because there are a few options on how to set it up the getting started guide leaves a bit to the imagination.

https://writefreely.org/start

So this post will outline the specific steps I took to setup my instance. Without further ado this is how to get started with WriteFreely on Debian 10.

Setting up the Debian Server

Digital Ocean has a ton of good tutorial posts for different development purposes. Most of this post is extracting the relevant parts from other tutorials found around the web. I'll be listing the links per section, I highly recommend checking them out for further context and info.

https://www.digitalocean.com/community/tutorials/initial-server-setup-with-debian-10

ssh into root on the server, we use root to setup a user.

ssh root@*server_ip*

Create a new user:

adduser *username*

Set this new user as sudo:

usermod -aG sudo *username*

Now we have to setup the firewall.

apt update
apt install ufw

List the different applications:

ufw app list

Now we need to allow OpenSSH:

ufw allow OpenSSH

And next we need to enable it:

ufw enable

This means the firewall will block everything besides ssh connections.

The next thing we have to do is copy the SSH key used to access root into the current userspace.

cp -r ~/.ssh /home/*username*
chown -R *username*:*username* /home/*username*/.ssh

Now you can ssh with your new user account:

ssh *username*@*server_ip*

The rest of this post assumes you are accessing the server via this new user account just made.

Installing MySQL

WriteFreely gives the option of using two database backends. MySQL and Sqlite. I chose MySQL because it seems to be the default. This was interesting for me because I've never used MySQL before (typically I use Postgres). I sourced the relevant info for this section from this site:

https://tecadmin.net/install-mysql-on-debian-10-buster/

There doesn't seem to be a package in the official Debian repository for MySQL, but MySQL as an organization does provide their own custom deb we can use.

We download the deb with wget, and use dpkg (the low level package tool for Debian) to manually install the deb package.

wget http://repo.mysql.com/mysql-apt-config_0.8.13-1_all.deb
sudo apt install gnupg # Needed dependency
sudo dpkg -i mysql-apt-config_0.8.13-1_all.deb

Now we can use apt to finish the installation process. The difference between apt and dpkg, is that apt is functionally a high level wrapper around dpkg.

sudo apt update
sudo apt install mysql-server

This will ask you to setup a password and what version of mysql you want to use. I wasn't able to get version 8.0 to work with writefreely, so make sure to choose mysql version 5.

WARNING: Make sure to do this step correctly because it is very hard to undo. I was forced to destroy and reset my droplet because I couldn't figure out how to reverse this.

Now that the mysql server is installed we can access the mysql server like so:

mysql -u root -p

Here we create a new database:

CREATE DATABASE writefreely;

We also want to make a new user:

CREATE USER '*username*'@'localhost' IDENTIFIED BY '*password*';

Now we grant privileges to the new user:

GRANT ALL PRIVILEGES ON *.* TO '*username*'@'localhost';

Make sure to update the privileges:

FLUSH PRIVILEGES;

Installing Nginx

Not going to Nginx kind of scares me, luckily Digital Ocean once again has a great tutorial, and WriteFreely provides an example config to use. So it was just a matter of connecting the dots.

https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-debian-10

The digital ocean tutorial has a lot of extra information that is helpful for general use cases, but not entirely relevant for what we are doing. So I'll outline the specific useful parts here.

Install nginx like so:

sudo apt install nginx

Easy. Now we have to get our environment all nice and tidy. We do this by setting up a server block.

First we create a new directory to host our files for the site:

sudo mkdir -p /var/www/*your_domain*/

Now we set permissions for this directory to our user account:

sudo chown -R $USER:$USER /var/www/*your_domain*/

This directory is where we will be installing the WriteFreely app later.

Now to configure the server block:

sudo nano /etc/nginx/sites-available/*your_domain*

Copy this config, and change the relevant details (most specifically the example.com url):

server {
    listen 80;
    listen [::]:80;

    server_name example.com;

    gzip on;
    gzip_types
      application/javascript
      application/x-javascript
      application/json
      application/rss+xml
      application/xml
      image/svg+xml
      image/x-icon
      application/vnd.ms-fontobject
      application/font-sfnt
      text/css
      text/plain;
    gzip_min_length 256;
    gzip_comp_level 5;
    gzip_http_version 1.1;
    gzip_vary on;

    location ~ ^/.well-known/(webfinger|nodeinfo|host-meta) {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_pass http://127.0.0.1:8080;
        proxy_redirect off;
    }

    location ~ ^/(css|img|js|fonts)/ {
        root /var/www/example.com/static;
        # Optionally cache these files in the browser:
        # expires 12M;
    }

    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_pass http://127.0.0.1:8080;
        proxy_redirect off;
    }
}

We enable this server block by creating a symbolic link to the sites-enabled directory:

sudo ln -s /etc/nginx/sites-available/*your_domain* /etc/nginx/sites-enabled/

In the digital ocean tutorial, they also recommend uncommenting a value in etc/nginx/nginx.conf. However, this step may be optional. I did it anyways just to be safe.

Find a line that looks like this:

# server_names_hash_bucket_size 64;

Uncomment it. Then restart nginx:

sudo systemctl restart nginx

We are almost done with nginx configuration and one step closer to our beautiful blog. The last thing we have to do before setting up the writefreely app is securing nginx with lets encrypt.

Securing Nginx with Let's Encrypt

Once again, these steps are taken from a great tutorial created by Digital Ocean.

https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-debian-10

Install certbot like so:

sudo apt install python3-acme python3-certbot python3-mock python3-openssl python3-pkg-resources python3-pyparsing python3-zope.interface
sudo apt install python3-certbot-nginx

Certbot is ready to go, and we will use it setup our certificates and integrate them into nginx.

Before we do that, we first have to allow https traffic through the firewall.

sudo ufw allow 'Nginx Full'

Now we can obtain our certificate:

sudo certbot --nginx -d *your_domain*

If you want the www prefix for your site it is like this:

sudo certbot --nginx -d *your_domain* -d www.*your_domain*

That has a few config steps to go through and your set. We can test the auto renewal feature like this:

sudo certbot renew --dry-run

Installing WriteFreely

At last, the moment we've been waiting for. If you made it this far, good job, you're a trooper.

We begin by downloading the latest release of writefreely, which can be found here:

https://github.com/writeas/writefreely/releases/

Copy the link location for the latest release and then use wget to download it to the server.

cd /var/www/*your_domain*
wget *link*
tar -xzvf *downloaded_tar_file*

Now you should see the writefreely binary. We use that binary to setup the config:

./writefreely --config

It will run through a few steps asking how you want it configured. Since we used Nginx select the option for production with reverse proxy. Make sure to set the database name, username, and password as the matching fields we used earlier.

I also selected it to be a single user blog, and I enabled federation. The rest of the defaults seem to work fine as well.

If this fails, it is most likely because you made a mistake with the MySQL configuration.

The next step is to setup the keys:

./writefreely --gen-keys

Now we are almost ready to fire up the website. Before we do that, we want to make the writefreely binary a systemd service. WriteFreely provides this great example config which should be placed at /etc/systemd/system/writefreely.service:

[Unit]
Description=WriteFreely Instance
After=syslog.target network.target

[Service]
Type=simple
StandardOutput=syslog
StandardError=syslog
WorkingDirectory=/var/www/example.com
ExecStart=/var/www/example.com/writefreely
Restart=always

[Install]
WantedBy=multi-user.target

We kickstart the service:

sudo systemctl start writefreely

Verify that everything is working correctly:

sudo journalctl -f -u writefreely

If there are no errors then you did it!

Now navigate to your site with the browser, and voila! You have a self hosted, free software, federated blog running on Debian. I don't know about you, but that has got me excited.

Conclusion

Hopefully someone in the world finds this useful. I certainly would've liked to have been spoonfed these instructions. This was quite the technical tutorial, but it is my intent to post about different sorts of things. Probably mostly about tech, but hey, that's what I know!

This is fancycade, signing off.