This is a follow up post to my introductory post of wing. wing is the blog engine I use to build the blog you are reading right now. I've been working on wing off and on for about 2 years now. Over that time there have been incremental improvements, but the basic flow is pretty much the same. In this post I'll highlight a few different improvements.
For a while my atom feed was displaying markup and not html, which most rss readers don't know how to read. There were also some other specific fields that were missing which I found when working on separate rss related project.
When a user runs wing publish a prompt will appear with a list of unpublished prompts. To publish a
post a user simply inputs a number associated with the post. This is nice because it reminds you of which
posts you may have been working on. This is a major improvement over inputting the file name of the markup post. That turned out to be error prone and inspired the simpler mechanism that exists today.
If a post gets published by accident for whatever reason you can simply run wing revert and it will remove the latest post from your local posts database (aka 'posts.scm').
I've overhauled the tests to run a full integration test over all the commands starting from a nonexistent project.
The one other user I've had that has messaged me said they ran into trouble getting started with wing.
I had seemed to have missed the edge case where the posts.scm file doesn't exist. Thus making it impossible to actually start a new blog.
This has been fixed and it is very easy to get started with wing now.
This is the feature I'm most excited about. Or at least the one that took the most work, and I am quite proud of how it turned out.
This feature is partially to blame for giving me a great excuse to procastrinate writing blog posts. We shall see if I have broken my procrastination streak.
What happened is that I noticed when building my small amount of posts the result was not instant. I thought this was odd since chicken is decently fast. If it was taking 1 second for only a handful of posts, how was this going to be long term?
I've heard often how jekyll can begin to be noticeably slow for a large number of posts. This wasn't really acceptable to me. However, keep in mind, everything was working fine. The time it took was 1 second.
What I'm trying to tell you is that I didn't need to go on this yak shave, but I did it anyways.
After some ad hoc benchmarking I eventually found the culprit. lowdown is a markdown to sxml library written in chicken. This makes installing it quite easy if you can install chicken packages (eggs). However, this one function was responsible for the vast majority of time spent running the program.
The fix was to replace lowdown with a commonmark wrapper library. Fortunately, there already was a commonmark egg written in Chicken. Unfortunately it didn't have markdown to sxml support. As this is one of the key features of wing, I had not choice but to implement this myself.
The nice part about this is that I was able to write it all in Chicken and not C.
The amount of work this was going to take really made me question how worth it all of this was.
yet I persisted. The thought of this yak shave not only made me laugh, but the challenge excited me.
I also guilted myself by telling myself I couldn't write any more posts until I had this done. I'm not sure why this was effective, because as I've said before, it takes a lot of motivation for me to write.
Essentially the process of writing this function involved converting an in memory tree like representation of markdown into nicely formatted sxml that also looked kind of like html (since the sxml needs to be converted into html in the final step).
As you can imagine this is a complicated and edge case prone process. Quickly I realized it was important for me to be building against a wide set of test cases. Luckily, there is the markdown testsuite. This was a literally tricky in my case, because my output wasn't html, it was sxml. So I had to sort of translate this all into sxml cases too. lowdown was exactly a great help, because it provided a reference point for how my sxml was supposed to look like. As it turns out. Using lowdown only got me so far. It became apparent that lowdown actually failed a few of these test cases. My idea of 1 to 1 with lowdown was shot. However, making a better implementation is not a bad place to end up.
After a few months I eventually passed all 128 test cases. Since the amount of code I had to write was more than the existing egg, I ended up publishing a new egg for cmark.
Once this egg was accepted in the coop. I was able to easily integrate it into wing as a dependency to replace lowdown.
The eventual performance increase I got was 10x. I think its possible to improve lowdown's performance, but its likely the work would not be trivial. Leveraging a well tested native library is fine in my book. Now when I run wing build it is instant.
So yeah I went on a crazy yak shave to improve the performance of a program that only took 1 second in the first place. I'm still testing out wing, but I think I'm pretty close to feeling good about a 1.0.0 release.
Content for this site is CC-BY-SA.