Have you ever despaired in front of your computer waiting for a build after a minor change? Did you ever swear to all gods in that process? I did, and I usually do. Being “southern” and hot-blooded is a problem sometimes. But it also gives me the motivation to find a way to improve that.
Written by Jordi Pujol
Let’s define the scenario
I’ve been working in a project with a codebase like this:
- 2 common libraries, that can be built from time to time and installed. About 1k cpp files to build
- Main application codebase depending on those libraries with about 3k cpp files to build
- Cmake based build system
- Linux / gcc / QtCreator
The first thing I did was collecting all low hanging fruits: I used ninja as build generator, that saved me some precious time on re-building (as easy as add –GNinja on the Cmake call) and I’ve activated ccache on my kit (manage kits and choose the ccache-based gcc version I wanted to use)
But that was just a minor improvement. I needed more speed and I wanted it also when doing full rebuilds. That was impossible only with those tools, but I had a card hidden in my sleeve.
I suspect I may suffer from a slight case of hoarder syndrome, as I tend to accumulate a bunch of old computers just because I think they may be useful to me in the future. In fact, I have an old machine, that used to be my HTPC, configured to take care of my home automation control system (I installed Home Assistant there just for the fun of it).
But I still had 3 more laptops dusting around that I could use:
- Lenovo B50-10 (Pentium N3540, @2.16GHz, 2 cores, 64 bits)
- HP Pavilion 15 b123es (AMD Dual-Core A4-4355M, @ 1.9GHz)
- Clonic laptop (i7-4700MQ CPU @ 2.40GHz, 12Gb memory)
The most powerful of these is my old i7 laptop. It’s now 10 y/o but yet a powerful machine. It’s usually idling around and I use it on the rare occasions I need to do some windows stuff, so it was just sitting close to me, almost useless.
So, why don’t I use it as a build server on a farm? I dug a little bit and the easiest way to test this was installing icecc, very easy to configure and set up. I did some trial and error before I got the right combination. Initially, I installed it dockerized in my windows machine, but it was unreachable from outside, so no way. Later on, I tried installing it inside WSL. I had to “hack” its auto-startup and I had to log in my user to have it active. But it caused some weird failures while building. Finally, I just made it dual-boot with a ubuntu 20.04 server install and things were smoother.
Setting up the extra CPUs
My final setup is this one:
- XPS 15 laptop => my usual dev machine & build server (i7-6700HQ CPU @ 2.60GHz, 16Gb)
- old i7 laptop => build server (slave)
- Lenovo laptop => build scheduler
I only used one of the 2 less powerful laptops as scheduler. Each one was giving me only 2 extra CPUs and the overhead of managing them wasn’t worth the effort.
And here are the numbers I got. Impresive, right?
Adding a second machine, with more or less same computing power, make my builds 1,76 times faster. Not a total 2x, because of network overhead and different CPU, and also because all preprocessing and linking happens in my dev machine, but good enough
In summary: I’ll keep this configuration up & running at home, as it makes me waste less time doing my daily work and caused me 0 trouble to do it. Win-win!
And here's how to do it!
If you are still reading, here comes the “how to” section.
Installing icecc was pretty straightforward in my Ubuntu machine, just a sudo apt-get install icecc icecc-monitor and you’re almost done. As you may note, I installed the monitor to check if the machines were doing good usage of the CPUs. In the other 2 machines, I just installed icecc.
Setting it up in all 3 computers was quite easy, just edit the file /etc/icecc/icecc.conf and set up this:
ICECC_NETNAME=”<choose a farm name>”
In my case, I don’t even have a local DNS server, so I simply set up a fixed IP for the scheduler and wrote the same IP in all 3 machines
Of course, I set up icecc (and the scheduler, in the one that had it) to start up the service automatically with sudo service enable icecc / icecc-sceduler, this way I don’t have to think about starting it every time I power the machines on.
Don’t forget to set up this env var CCACHE_PREFIX=icecc so, you don’t have to bother about chaining icecc with ccache. It’s also the recommended way ccache maintainers want it to be done. You will avoid a lot of trouble and messages about icecc calling itself recursively.
A hint: to avoid the laptops to enter “suspend mode” when closing the lid, I had to modify the file /etc/systemd/logind.conf and change the following lines :
Now, after the experiment suceeded, my next dream is buying a cheap spare datacenter server (I found for about 400€ a Dual xeon, with 24 threads available, 48Gb RAM and 2x2Tb 7.5k sata HD) and install that in my basement. The fun never ends!