When I last blogged here, I was spending a large amount of my free time on the DevOpsChops side project. The project was taking shape incredibly quickly and I could see myself releasing it in a month or two with the progress being made. Since then, I’ve spent approximately zero time on it.
The original culprit was winning a Nintendo Switch. I considered giving it away, but first I just had to play Breath of the Wild as it was heralded as one of the top games of all time. Turns it it’s also a very time-consuming adventure to complete. That ate up almost a month of time. After that I somehow convinced myself that I needed to play some classic PC games as a form of “research” into improving the game aspect of DevOpsChops. Before I knew it I’d worked through the entire BioShock series (all excellent), a number of puzzle games like Braid and Talos Principle, and even a few VR games.
I don’t regret that I played these games but I do regret that I haven’t spent any time on DevOpsChops. There are so many great video games out there and you could spend all your spare time playing them. They seemed to make me lazier, by providing big in-game rewards for straightforward and engaging goal-oriented work.
Worse, they seemed to sap my creative drive and made me despair of making something cool myself, as the level of craftmanship in games like BioShock is just awe-inspiring making any indie effort look basic. I need to recognize that few experiences are going to compare to playing one of the top reviewed games ever produced by a large, renowed studio. And that’s fine! There’s a galaxy of space for all sorts of creations to exist and thrive in.
There’s a regular pattern that myself and many of my friends fall into. We get obsessed by a particular side project or field of study for a while, before something else catches our fancy. In some ways this is an advantage; in the fast moving world of software it’s good to always be experimenting with new technologies, and if a fixation is not particularly healthy (such as video games) it never seems to last too long.
But it also means that my hard drive and notebooks are full of half-finished projects. When I look through some of these projects, I think, it’s not perfect but people would have liked it, I should have just released it! I’m determined not to let DevOpsChops be one of those projects.
While developing websites I test in both Chromium and Firefox. I recently ran into a problem where VM launch requests from the DevOpsChops website frontend would always work in Chromium, but would occasionally fail in Firefox. A VM launch request here is an AJAX POST request where a user clicks a button, then waits 3-5 seconds for a VM to be launched and then they get a shell on the VM.
Specifically, the error appearing in the browser console was: “The Same Origin Policy disallows reading the remote resource at http://127.0.0.1:8888/. (Reason: CORS request did not succeed). Status code: (null)”
I instinctively searched for the wording of the error, and all the answers were about broken CORS policies. I soon realized that the error I was seeing wasn’t really to do with CORS. It was only reported as a CORS failure since it was a complex request (POST request with JSON body) that never got a response. The real issue was not receiving a response from the server.
So why was I getting a server response in Chromium but not in Firefox? A clue was that this error occurred more often when my CPU was busy. This was related to timing in some way.
The solution
Further research led to a Mozilla Bugzilla report “Empty response from POST AJAX request from Datatables”, where similar behaviour was occurring on POST requests to an interactive tables site, although the response was empty rather than throwing a CORS error. The root cause is that Firefox has a configuration setting network.http.network-changed.timeout which is set to 5 by default. Network requests which are inactive for over 5 seconds are killed.
When I increased this setting in about:config I no longer faced the error. My long-term fix has been for the DevOpsChops server to keep the connection alive by pushing partial responses until the full response can be sent.
Future of Firefox
But, it does make me think that behaviour like this is part of the reason for Firefox’s decline. Chrome is more tolerant of badly-written websites and the problem with DataTables was never resolved. Many Firefox users won’t bother to figure out why a website won’t work and will simply switch to Chrome. Some technical websites like KodeKloud recommend only using Chrome because it’s easier for them than spending the effort on making sure their site is fully compatible.
The command line presents an interactive environment which can be adapted to games quite easily. A lot of old-school text adventure games like Zork and MUDs were based on the command line. This post focusses rather on games which teach you GNU/Linux-centric command line knowledge in the process of playing.
A classic series of command line challenges on a server that is accessed over SSH. User permissions are the mechanism used to restrict access to each level. I put the question mark after 2012 as apparently the original game has been around a lot longer.
Pros: Fun variety of challenges that escalate in difficulty and encourage self-study and exploration. Additional wargames dive deeper into areas like binary exploitation.
Cons: Aimed at absolute beginners but some will find just the SSHing in and challenge conventions confusing. The password mechanism for accessing the next level is somewhat clunky.
There’s been a murder, and you’re a detective. The evidence is spread out among multiple text files. You have to write text wrangling and filtering commands to narrow down the culprit. There’s also a similar murder mystery for SQL.
Pros: Great theme, fun premise. Multiple ways to solve. Hints and a cheatsheet are provided for less experienced players.
Cons: Too short, can be solved in less than 10 minutes if you know your grep. Could have been designed so that unveiling the final answer was more satisfying.
This is a series of mini command line tasks that you have to perform, e.g. Print the relative file paths, one path per line for all filenames that start with "access.log" in the current directory.
Pros: Fast UI and it’s addictive. Alternative solutions are offered which are interesting to read.
Cons: Uses a non-interactive shell with one-shot Docker images rather than providing a realistic environment. Some valid solutions don’t work. Most challenges are easy and it doesn’t scale up to a substantial level of difficulty by the end.
This is a dungeon crawler game where each room is a directory, objects inside the room are files, and interactive objects are shell scripts. So you naturally have to use command line tools to explore the dungeon.
Pros: The idea of mapping a filesystem to a game environment is a strong one. This feels more like an actual game than any of the others here, and simple commands are learned quickly by repetition.
Cons: Some of the instructions, in trying overly to be noob-friendly (“drag and drop this folder into your terminal”) manage to cause more confusion. Tracking game state is done manually perhaps to teach environment variables but this gets tedious. The game is only for beginners; if you know too much it’s easy to break.
Billed as “like LeetCode for Linux”, this contains a number of individual timed troubleshooting challenges that are similar to hands-on SRE interviews.
Pros: Realistic scenarios that make debugging fun. Not aimed at beginners, the challenges touch on advanced topics. Challenges require you to use tools that help you understand Linux at a deeper level, which you rarely get outside of a real-life context.
Cons: The UI isn’t great and you have to wait a while for instances to launch, which spoils the immersion. Some of the challenges are obscure and don’t clearly communicate a teachable idea.
Suggested order
Overall, there’s strong and weak points to each of these games. For complete beginners, Bashcrawl is a great way to start, and after some fluency on the command line has been reached, Command Challenge and Bandit could be used to expand one’s repertoire. After that, Command Line Murders could be considered a fun “final boss” to the Command Challenge. Finally, SadServers is less game-like but still exemplifies the more difficult end of the spectrum of acquiring Linux skills through play.
DevOpsChops is a game that I’m building based on the command line. You might need to fullscreen the video to see the demo clearly:
There are number of self-contained challenges which rapidly build up fluency in Linux, networking, cloud, databases and related areas. Challenges on high-level DevOps concepts are also on the roadmap.
On the backend, the VMs are running on Firecracker, so they launch fast and are isolated from the host system. Much of the work so far has been in creating a VM templating system that enables challenges to be fully specified in a few lines of configuration.
I think that existing DevOps learning resources are pretty dull on average. They tend to overwhelm learners with the massive amount of tools and teach them in a lecture style. What is lost is the idea of play, where you are motivated to explore and seek out answers for your own problems. Memorising details is unimportant compared to knowing how to diagnose an issue and how to find a solution. Linux, Kubernetes, AWS are complex but (mostly :D) reasonable systems that you can gain an intuition for after enough hands-on experience.
So that’s why I refer to this project as a game. It borrows ideas from CTFs, in transforming what would otherwise be learning abstruse material into a competitive, social, engaging experience. While fixing systems may be less exciting than pwning them, I aim to make up for this with a sleek platform and well-written content which slowly escalates in difficulty until players find themselves debugging realistic scenarios.
Software engineers in general say they wish they knew more about these topics. Ultimately, DevOpsChops is an attempt to lower the activation energy and build something which is simply fun to play.
