Fully automating the daily developer humdrum

In the beginning, and every day, the toil

Do you remember those first few days when you joined a new software project? Perhaps it was at your current job? Or maybe when you started contributing to an open source project? Or maybe it was when you decided to get your hands dirty with the latest fad framework?

Even in 2017, the place you often start is by going through a list of step-by-step instructions in a wiki, a README, or given to you by a fellow developer. Usually, you need to install this, then install that, put this file here, and run this script there.  It’s pretty tedious, error-prone, and often requires improvisation to account for things changing.

And of course, it’s not just at the start of a project. There are tedious tasks that you have to perform daily as a developer. Building your application, running unit tests, running functional tests, setting up seed data, creating certificates, generating versioned packages, setting up infrastructure, deploying your application, the list goes on.

The developer automation industry

Alas, for software developers, setting up and managing environments, and the daily humdrum involved in development, testing and production has been a timeless problem. Automating all of this has produced whole products, companies, and ecosystems. There are categories and sub-categories of tools and services to automate all aspects of the day-to-day of a developer.

Docker graces us with its glory

Along came Docker. Docker, and containers in general, are a giant leap forward to making all of this easier. Create a Docker image or two with all of your application components, dependencies and configuration, and you can be assured that you can spin up a consistent environment for your application. Once you’ve gone through the work of creating the images, spinning up containers is quick, easy, and consistent on most modern infrastructure. Composing and managing multiple containers for more complex environments is also straightforward with the emergence of solid orchestration solutions (Kubernetes is the way forward here).

The adoption trends for containers indicate that many see the glory. Most cloud providers offer a container solution, and all indications are that enterprises are starting to use containers for managing environments, both in testing and production (by the way, I know that not everyone will share the container love – by all means, vent in the comments below).

The curious case of local development environments

Indeed, even most cloud CI (continuous integration) providers these days are built on containers. Yes, from automated CI builds, to testing, to production, containers are the answer for consistent environments, and indeed automation for development.

Despite the adoption of containers for testing and production, we generally don’t seem to use containers for creating consistent local development environments. Of course, I am painting with a broad brush here but the usual story for setting up a local development environment remains read the wiki, or read the README, and go through a step-by-step list of things to install, scripts to run, and configuration to setup.

Ready to develop the team’s web application? Install JDK 8, and Maven, and run the Maven build script to build the backend. Install Node.js 8, change the configuration in the package.json, and run npm install, and  npm run start to build and run the frontend. Install MySQL, and Rabbit, change the configuration on the backend to point to them, and then run  mvn spring-boot:run to run the application.

This can all obviously be made into a Docker image so that running the app is just launching a container from that image. Or perhaps a set of containers, composed with Swarm or Kubernetes. Similarly, a Docker container based on an image already containing Maven can be used to build the backend. And a Docker container based on a Node.js image can be used to build the frontend. But we don’t do this, generally speaking. Kudos if you do – whether you do or don’t, read on.

Containers for local development

So why don’t we use a Docker container to do things like compiling the application, or running functional tests? Well, consider what you would need to do in order to use Docker to perform these day-to-day development tasks:

  • You’ll have to first install Docker
  • Then, you’ll have to create a Dockerfile with the tools you would need to perform your development task (or search for a Docker image which has what you need)
  • Then you would have to write a script which would build an image with the Dockerfile, and then have that script launch a Docker container with that image
  • And you would have to do that for each of the various development tasks you may have to perform.

Of course, someone on your team could go through the trouble of creating the Dockerfiles and scripts for your team (or turn it into an internal framework or tool), and check them in to your project repository. Maybe you already do this to automate your team’s development tasks. Kudos if you do – whether you do or don’t, read on.

 The pesky last mile of manual installation

Even if you have automated it all, using scripts and Docker containers, there is still that last step of installing Docker itself. Even with all that automation in your project’s Git repository, the next person will still need to install Docker on their machine first before being able to automate their tasks. And that’s only if you invested in automating the rest of it using containers.

Enter Sandbox

Here’s where we throw our hat into the developer automation ring. With Sandbox, we are hoping to the move the needle slightly further with developer automation. Among the ways we hope to do this is by removing this last little bit of manual installation holding developers back from full automation in creating consistent environments.

How? Well, Sandbox is a set of tiny cross-platform binaries: all together, these binaries are under 200KB. They are designed to be checked in to your project repository. Why does that make a difference? Well, it means you can check out your project’s Git repository, and have Sandbox immediately available to run, and automate your development tasks! So imagine checking out your repository and running functional tests:

git checkout https://github.com/stackfoundation/sbox-mean
cd sbox-mean
./sbox run functional-tests

Because Sandbox is committed to your repository, it’s available when you check out the repo, and that’s all you need to run your tests. You don’t need to install any software before running the sbox command – not even Docker.

Again, you can do this because Sandbox is tiny and designed to be committed into your repository. And you can do this because Sandbox is cross-platform – it works on Windows, macOS, and most Linux distributions (we hope to add more soon). And you can do this because what Sandbox runs are workflows, where each step in the workflow runs a Docker container. We think this is why Sandbox can fully automate development, including that last stretch. Sandbox let’s you run a fully automated development workflow straight from Git.

Built on Docker and Kubernetes

So, how do we fit something that runs containers into something tiny enough to be checked into your Git repository? As you might imagine, when you run Sandbox on a machine for the first time, it downloads additional components that makes it all work. Under the covers, Sandbox is built on Docker and Kubernetes. When you run Sandbox for the first time, it downloads enough to setup and run a single-node Kubernetes cluster. Of course, once the cluster is setup, you have full access to the Docker daemon and Kubernetes cluster (check out the cluster management section of the Sandbox documentation for more). A side effect of all this is that Sandbox is the simplest way to get Docker & Kubernetes running on your local machine.

Details on all of this is described in the Architecture section of the Sandbox documentation.

We need your help

When you start with the goal of creating something that works across platforms, you quickly realize you’ve signed up for a pretty big challenge. When your goal is to deal with all the platform nuances involved in starting up a single-node Kubernetes cluster, using the best hypervisor available on that platform, you make the challenge even harder. You’ve heard us promise big things with Sandbox, and we want to deliver. But we’re a small team and so we need your help, especially now while Sandbox is still in Alpha.

If you are developer, we would love it if you could try running a simple workflow with Sandbox, and tell us how it goes. If Sandbox fails to run on your machine, we especially want to hear from you. But we want to hear from you even if it all works as it was designed to. Leave a comment on the blog, tweet at us, file an issue on our GitHub issue tracker, find us on Gitter, or drop us an email. We want to hear what you think.