DevOpsChops Diaries

Procrastination

2023-07-02T00:00:00+00:00

When I last blogged here, I was spending a large amount of my free time on the DevOpsChops side project. The project was taking shape incredibly quickly and I could see myself releasing it in a month or two with the progress being made. Since then, I’ve spent approximately zero time on it.

The original culprit was winning a Nintendo Switch. I considered giving it away, but first I just had to play Breath of the Wild as it was heralded as one of the top games of all time. Turns it it’s also a very time-consuming adventure to complete. That ate up almost a month of time. After that I somehow convinced myself that I needed to play some classic PC games as a form of “research” into improving the game aspect of DevOpsChops. Before I knew it I’d worked through the entire BioShock series (all excellent), a number of puzzle games like Braid and Talos Principle, and even a few VR games.

I don’t regret that I played these games but I do regret that I haven’t spent any time on DevOpsChops. There are so many great video games out there and you could spend all your spare time playing them. They seemed to make me lazier, by providing big in-game rewards for straightforward and engaging goal-oriented work.

Worse, they seemed to sap my creative drive and made me despair of making something cool myself, as the level of craftmanship in games like BioShock is just awe-inspiring making any indie effort look basic. I need to recognize that few experiences are going to compare to playing one of the top reviewed games ever produced by a large, renowed studio. And that’s fine! There’s a galaxy of space for all sorts of creations to exist and thrive in.

There’s a regular pattern that myself and many of my friends fall into. We get obsessed by a particular side project or field of study for a while, before something else catches our fancy. In some ways this is an advantage; in the fast moving world of software it’s good to always be experimenting with new technologies, and if a fixation is not particularly healthy (such as video games) it never seems to last too long.

But it also means that my hard drive and notebooks are full of half-finished projects. When I look through some of these projects, I think, it’s not perfect but people would have liked it, I should have just released it! I’m determined not to let DevOpsChops be one of those projects.

CORS Request Failing in Firefox but not Chrome

2023-04-06T00:00:00+00:00

The error

While developing websites I test in both Chromium and Firefox. I recently ran into a problem where VM launch requests from the DevOpsChops website frontend would always work in Chromium, but would occasionally fail in Firefox. A VM launch request here is an AJAX POST request where a user clicks a button, then waits 3-5 seconds for a VM to be launched and then they get a shell on the VM.

Specifically, the error appearing in the browser console was: “The Same Origin Policy disallows reading the remote resource at http://127.0.0.1:8888/. (Reason: CORS request did not succeed). Status code: (null)”

I instinctively searched for the wording of the error, and all the answers were about broken CORS policies. I soon realized that the error I was seeing wasn’t really to do with CORS. It was only reported as a CORS failure since it was a complex request (POST request with JSON body) that never got a response. The real issue was not receiving a response from the server.

So why was I getting a server response in Chromium but not in Firefox? A clue was that this error occurred more often when my CPU was busy. This was related to timing in some way.

The solution

Further research led to a Mozilla Bugzilla report “Empty response from POST AJAX request from Datatables”, where similar behaviour was occurring on POST requests to an interactive tables site, although the response was empty rather than throwing a CORS error. The root cause is that Firefox has a configuration setting network.http.network-changed.timeout which is set to 5 by default. Network requests which are inactive for over 5 seconds are killed.

When I increased this setting in about:config I no longer faced the error. My long-term fix has been for the DevOpsChops server to keep the connection alive by pushing partial responses until the full response can be sent.

Future of Firefox

But, it does make me think that behaviour like this is part of the reason for Firefox’s decline. Chrome is more tolerant of badly-written websites and the problem with DataTables was never resolved. Many Firefox users won’t bother to figure out why a website won’t work and will simply switch to Chrome. Some technical websites like KodeKloud recommend only using Chrome because it’s easier for them than spending the effort on making sure their site is fully compatible.

5 Games for Learning Linux and the Command Line

2023-03-28T00:00:00+00:00

The command line presents an interactive environment which can be adapted to games quite easily. A lot of old-school text adventure games like Zork and MUDs were based on the command line. This post focusses rather on games which teach you GNU/Linux-centric command line knowledge in the process of playing.

OverTheWire Bandit (2012?)

Link

A classic series of command line challenges on a server that is accessed over SSH. User permissions are the mechanism used to restrict access to each level. I put the question mark after 2012 as apparently the original game has been around a lot longer.

Pros: Fun variety of challenges that escalate in difficulty and encourage self-study and exploration. Additional wargames dive deeper into areas like binary exploitation.

Cons: Aimed at absolute beginners but some will find just the SSHing in and challenge conventions confusing. The password mechanism for accessing the next level is somewhat clunky.

Command Line Murders (2013)

Link

There’s been a murder, and you’re a detective. The evidence is spread out among multiple text files. You have to write text wrangling and filtering commands to narrow down the culprit. There’s also a similar murder mystery for SQL.

Pros: Great theme, fun premise. Multiple ways to solve. Hints and a cheatsheet are provided for less experienced players.

Cons: Too short, can be solved in less than 10 minutes if you know your grep. Could have been designed so that unveiling the final answer was more satisfying.

Command Challenge (2017)

Link

This is a series of mini command line tasks that you have to perform, e.g. Print the relative file paths, one path per line for all filenames that start with "access.log" in the current directory.

Pros: Fast UI and it’s addictive. Alternative solutions are offered which are interesting to read.

Cons: Uses a non-interactive shell with one-shot Docker images rather than providing a realistic environment. Some valid solutions don’t work. Most challenges are easy and it doesn’t scale up to a substantial level of difficulty by the end.

Bashcrawl (2019)

Link

This is a dungeon crawler game where each room is a directory, objects inside the room are files, and interactive objects are shell scripts. So you naturally have to use command line tools to explore the dungeon.

Pros: The idea of mapping a filesystem to a game environment is a strong one. This feels more like an actual game than any of the others here, and simple commands are learned quickly by repetition.

Cons: Some of the instructions, in trying overly to be noob-friendly (“drag and drop this folder into your terminal”) manage to cause more confusion. Tracking game state is done manually perhaps to teach environment variables but this gets tedious. The game is only for beginners; if you know too much it’s easy to break.

SadServers (2022)

Link

Billed as “like LeetCode for Linux”, this contains a number of individual timed troubleshooting challenges that are similar to hands-on SRE interviews.

Pros: Realistic scenarios that make debugging fun. Not aimed at beginners, the challenges touch on advanced topics. Challenges require you to use tools that help you understand Linux at a deeper level, which you rarely get outside of a real-life context.

Cons: The UI isn’t great and you have to wait a while for instances to launch, which spoils the immersion. Some of the challenges are obscure and don’t clearly communicate a teachable idea.

Suggested order

Overall, there’s strong and weak points to each of these games. For complete beginners, Bashcrawl is a great way to start, and after some fluency on the command line has been reached, Command Challenge and Bandit could be used to expand one’s repertoire. After that, Command Line Murders could be considered a fun “final boss” to the Command Challenge. Finally, SadServers is less game-like but still exemplifies the more difficult end of the spectrum of acquiring Linux skills through play.

Announcing DevOpsChops

2023-03-23T00:00:00+00:00

DevOpsChops is a game that I’m building based on the command line. You might need to fullscreen the video to see the demo clearly:

There are number of self-contained challenges which rapidly build up fluency in Linux, networking, cloud, databases and related areas. Challenges on high-level DevOps concepts are also on the roadmap.

On the backend, the VMs are running on Firecracker, so they launch fast and are isolated from the host system. Much of the work so far has been in creating a VM templating system that enables challenges to be fully specified in a few lines of configuration.

I think that existing DevOps learning resources are pretty dull on average. They tend to overwhelm learners with the massive amount of tools and teach them in a lecture style. What is lost is the idea of play, where you are motivated to explore and seek out answers for your own problems. Memorising details is unimportant compared to knowing how to diagnose an issue and how to find a solution. Linux, Kubernetes, AWS are complex but (mostly :D) reasonable systems that you can gain an intuition for after enough hands-on experience.

So that’s why I refer to this project as a game. It borrows ideas from CTFs, in transforming what would otherwise be learning abstruse material into a competitive, social, engaging experience. While fixing systems may be less exciting than pwning them, I aim to make up for this with a sleek platform and well-written content which slowly escalates in difficulty until players find themselves debugging realistic scenarios.

Software engineers in general say they wish they knew more about these topics. Ultimately, DevOpsChops is an attempt to lower the activation energy and build something which is simply fun to play.

Games and the Future of Education

2023-03-20T00:00:00+00:00

I wrote up a summary on my main blog of a talk which makes great points about games design.

Communicating Between Firecracker microVMs using Linux Bridges

2023-03-16T00:00:00+00:00

The Firecracker doc on network setup works well for basic use cases. The doc shows you how to communicate between your host and Firecracker VMs, firstly by using a TAP device then in the advanced case by using a bridge device.

Up until now, I’ve only needed to use host-to-guest communication. But, I now want to enable VMs to network with each other. I created a bridge with its own /24, then attached two TAP devices to it, and launched microVMs configured with IPs in that subnet. Host-to-guest networking still worked, but networking between the VMs did not.

Unicasts not welcome here

More specifically, broadcast packets were being forwarded over the bridge, but not normal unicast packets. For instance I noticed that ARP requests (which are broadcasted to all hosts in a LAN) from one microVM were reaching the other VM. This was seen in the destination VM’s ARP table and in tcpdump. But the ARP replies which are unicasted from the receiving host to the requestor were getting dropped between the bridge and the requesting VM. This was odd because the packet MAC addresses matched the TAP devices and everything seemed to be configured correctly.

I found that nobody on the Internet seems to know how to fix this either. There’s an issue on the Firecracker GitHub and posts on StackExchange but no guide or hints anywhere as to how to get Firecracker-to-Firecracker communication working. In fact there’s lots of questions on StackExchange about this bridge behaviour in general, where only broadcasts make it through. Unfortunately the answer was usually that an iptables or arptables or ebtables rule was interfering, or that a particular sysctl setting had been overlooked (e.g. net.ipv4.ip_forward), none of which applied in my case.

Turning the bridge into a pseudo-hub

After discussing with a friend, I decided that an approach to try was to flush the MAC table of the bridge. Linux bridges behave like switches so with an empty MAC table it should flood all its ports until it knows on which port it can find a device.

Here’s the setup, 8f4b9e2676 is the bridge and 8f4b9e2676_1 and 8f4b9e2676_2 are TAP devices attached to microVMs with internally assigned IP addresses 10.0.80.2 and 10.0.80.3 respectively.

$ ip a
34: 8f4b9e2676: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 32:a4:c3:78:36:a7 brd ff:ff:ff:ff:ff:ff
    inet 10.0.80.1/24 scope global 8f4b9e2676
       valid_lft forever preferred_lft forever
    inet6 fe80::30a4:c3ff:fe78:36a7/64 scope link
       valid_lft forever preferred_lft forever
35: 8f4b9e2676_1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master 8f4b9e2676 state UP group default qlen 1000
    link/ether 56:ea:d1:d7:a1:f4 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::54ea:d1ff:fed7:a1f4/64 scope link
       valid_lft forever preferred_lft forever
36: 8f4b9e2676_2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master 8f4b9e2676 state UP group default qlen 1000
    link/ether 22:19:21:28:85:56 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::2019:21ff:fe28:8556/64 scope link
       valid_lft forever preferred_lft forever

With brctl you can see the MAC addresses of the interfaces connected to the ports of the bridge:

$ sudo brctl showmacs 8f4b9e2676
port no mac addr                is local?       ageing timer
  2     22:19:21:28:85:56       yes                0.00
  2     22:19:21:28:85:56       yes                0.00
  1     56:ea:d1:d7:a1:f4       yes                0.00
  1     56:ea:d1:d7:a1:f4       yes                0.00

A few handy guides say to use brctl setageing 0 in order to make the bridge act like a hub. But, I found it only worked to remove learned MAC addresses, not to remove the MAC addresses marked as yes under is local?. Still, it’s an important first step although the modern solution is to append ageing 0 to the ip link command when creating the bridge.

After further research, but there’s really not a lot of good resources on this, it’s possible to use the bridge fdb command to manipulate the routing table at a lower level:

$ sudo bridge fdb show | grep 8f4b9e2676
33:00:00:00:01 dev 8f4b9e2676 self permanent
00:5e:00:00:6a dev 8f4b9e2676 self permanent
33:00:00:00:6a dev 8f4b9e2676 self permanent
00:5e:00:00:01 dev 8f4b9e2676 self permanent
00:5e:00:00:fb dev 8f4b9e2676 self permanent
33:ff:78:36:a7 dev 8f4b9e2676 self permanent
33:00:00:00:fb dev 8f4b9e2676 self permanent
a4:c3:78:36:a7 dev 8f4b9e2676 vlan 1 master 8f4b9e2676 permanent
a4:c3:78:36:a7 dev 8f4b9e2676 master 8f4b9e2676 permanent
ea:d1:d7:a1:f4 dev 8f4b9e2676_1 vlan 1 master 8f4b9e2676 permanent
ea:d1:d7:a1:f4 dev 8f4b9e2676_1 master 8f4b9e2676 permanent
33:00:00:00:01 dev 8f4b9e2676_1 self permanent
00:5e:00:00:01 dev 8f4b9e2676_1 self permanent
33:ff:d7:a1:f4 dev 8f4b9e2676_1 self permanent
33:00:00:00:fb dev 8f4b9e2676_1 self permanent
19:21:28:85:56 dev 8f4b9e2676_2 vlan 1 master 8f4b9e2676 permanent
19:21:28:85:56 dev 8f4b9e2676_2 master 8f4b9e2676 permanent
33:00:00:00:01 dev 8f4b9e2676_2 self permanent
00:5e:00:00:01 dev 8f4b9e2676_2 self permanent
33:ff:28:85:56 dev 8f4b9e2676_2 self permanent
33:00:00:00:fb dev 8f4b9e2676_2 self permanent

And delete the routes that set the bridge as the master for the TAP devices:

$ sudo bridge fdb del 56:ea:d1:d7:a1:f4 dev 8f4b9e2676_1 master
$ sudo bridge fdb del 22:19:21:28:85:56 dev 8f4b9e2676_2 master
$ sudo brctl showmacs 8f4b9e2676
port no mac addr                is local?       ageing timer

It’s also necessary to configure the bridge-nf-call-iptables kernel parameter. This disables packets from passing over the bridge from being processed by iptables. I don’t understand why this needs to be done, since I removed all iptables rules and set all policies to ACCEPT and yet networking still didn’t work until I set it:

sudo sysctl -w net.bridge.bridge-nf-call-iptables=0

And now, amazingly, the inter-microVM communication works! From inside the 10.0.80.2 VM:

$ ping 10.0.80.3
PING 10.0.80.3 (10.0.80.3) 56(84) bytes of data.
64 bytes from 10.0.80.3: icmp_seq=1 ttl=64 time=1.37 ms

tcpdump of the bridge shows the packets going back and forth as expected. This feels like a massive hack and I’m not sure why it works when the correct MAC entries didn’t, but until I figure out what was going wrong originally, it will do the job. I only want to connect a maximum of two Firecracker VMs together, so the flooding won’t result in extra network traffic.