Have you ever despaired in front of your computer waiting for a build after a minor change? Did you ever swear to all gods in that process? I did, and I usually do. Being “southern” and hot-blooded is a problem sometimes. But it also gives me the motivation to find a way to improve that.
Written by Jordi Pujol
2023/5/19
Let’s define the scenario
I’ve been working in a project with a codebase like this:
- 2 common libraries, that can be built from time to time and installed. About 1k cpp files to build
- Main application codebase depending on those libraries with about 3k cpp files to build
- Cmake based build system
- Linux / gcc / QtCreator
The first thing I did was collecting all low hanging fruits: I used ninja as build generator, that saved me some precious time on re-building (as easy as add –GNinja on the Cmake call) and I’ve activated ccache on my kit (manage kits and choose the ccache-based gcc version I wanted to use)
But that was just a minor improvement. I needed more speed and I wanted it also when doing full rebuilds. That was impossible only with those tools, but I had a card hidden in my sleeve.
I suspect I may suffer from a slight case of hoarder syndrome, as I tend to accumulate a bunch of old computers just because I think they may be useful to me in the future. In fact, I have an old machine, that used to be my HTPC, configured to take care of my home automation control system (I installed Home Assistant there just for the fun of it).
But I still had 3 more laptops dusting around that I could use:
- Lenovo B50-10 (Pentium N3540, @2.16GHz, 2 cores, 64 bits)
- HP Pavilion 15 b123es (AMD Dual-Core A4-4355M, @ 1.9GHz)
- Clonic laptop (i7-4700MQ CPU @ 2.40GHz, 12Gb memory)
The most powerful of these is my old i7 laptop. It’s now 10 y/o but yet a powerful machine. It’s usually idling around and I use it on the rare occasions I need to do some windows stuff, so it was just sitting close to me, almost useless.
So, why don’t I use it as a build server on a farm? I dug a little bit and the easiest way to test this was installing icecc, very easy to configure and set up. I did some trial and error before I got the right combination. Initially, I installed it dockerized in my windows machine, but it was unreachable from outside, so no way. Later on, I tried installing it inside WSL. I had to “hack” its auto-startup and I had to log in my user to have it active. But it caused some weird failures while building. Finally, I just made it dual-boot with a ubuntu 20.04 server install and things were smoother.
Setting up the extra CPUs
My final setup is this one:
- XPS 15 laptop => my usual dev machine & build server (i7-6700HQ CPU @ 2.60GHz, 16Gb)
- old i7 laptop => build server (slave)
- Lenovo laptop => build scheduler
I only used one of the 2 less powerful laptops as scheduler. Each one was giving me only 2 extra CPUs and the overhead of managing them wasn’t worth the effort.
And here are the numbers I got. Impresive, right?
| Conditions | Library A | Library B | Main | Total | |
|---|---|---|---|---|---|
|
icecc inactive |
Items Start time End time Time spent |
883 13:30h 13:46h 00:16h |
633 13:46h 14:00h 00:14h |
3486 14:00h 15:49h 01:49h |
02:19h |
|
icecc active |
Items Start time End time Time spent |
883 12:02h 12:11h 00:09h |
633 12:11h 12:21h 00:10h |
3486 12:21h 13:21h 01:00h |
01:19h |
Adding a second machine, with more or less same computing power, make my builds 1,76 times faster. Not a total 2x, because of network overhead and different CPU, and also because all preprocessing and linking happens in my dev machine, but good enough
In summary: I’ll keep this configuration up & running at home, as it makes me waste less time doing my daily work and caused me 0 trouble to do it. Win-win!
And here's how to do it!
If you are still reading, here comes the “how to” section.
Installing icecc was pretty straightforward in my Ubuntu machine, just a sudo apt-get install icecc icecc-monitor and you’re almost done. As you may note, I installed the monitor to check if the machines were doing good usage of the CPUs. In the other 2 machines, I just installed icecc.
Setting it up in all 3 computers was quite easy, just edit the file /etc/icecc/icecc.conf and set up this:
ICECC_NETNAME=”<choose a farm name>”
ICECC_SCHEDULER_HOST=”192.168.1.20″
In my case, I don’t even have a local DNS server, so I simply set up a fixed IP for the scheduler and wrote the same IP in all 3 machines
Of course, I set up icecc (and the scheduler, in the one that had it) to start up the service automatically with sudo service enable icecc / icecc-sceduler, this way I don’t have to think about starting it every time I power the machines on.
Don’t forget to set up this env var CCACHE_PREFIX=icecc so, you don’t have to bother about chaining icecc with ccache. It’s also the recommended way ccache maintainers want it to be done. You will avoid a lot of trouble and messages about icecc calling itself recursively.
A hint: to avoid the laptops to enter “suspend mode” when closing the lid, I had to modify the file /etc/systemd/logind.conf and change the following lines :
HandleLidSwitch=ignore
HandleLidSwitchExternalPower=ignore
…
LidSwitchIgnoreInhibited=no
UPDATE: after the experiment suceeded, my next dream was buying a cheap spare datacenter server (I found for about 600€ a Dual xeon, with 24 threads available, 64Gb RAM and 4x1Tb 7.5k hot-swap sata HD in a RAID5 config) and installed that in my basement. The fun never ends!
I’ve updated my hardware, and upgraded my laptop too: I’ve duplicated the memory to 32Gb and also changed the NVMe disk to another one faster that has 1Tb capacity. Everything helps to win some speed!
The 2 old laptops have been retired in favour of this server, that hosts some virtual machines and a linux SQL server instance and stays idleing most of the time. This server acts as scheduler and as a node, as it’s powerful enough to just give me the peak I need when I’m building at home. Numbers come down here
| Conditions | Library A | Library B | Main | Total | |
|---|---|---|---|---|---|
|
icecc inactive |
Items Start time End time Time spent |
883 16:07h 16:12h 00:05h |
644 16:12h 16:18h 00:06h |
3623 16:18h 17:17h 00:59h |
01:10h |
|
icecc active |
Items Start time End time Time spent |
883 17:51h 17:54h 00:03h |
644 17:54h 17:57h 00:03h |
3623 15:57h 18:34h 00:37h |
00:43h |
They are quite surprising: There is an improvement, but a little bit less difference than before with icecc active/inactive. In this case the reason is that I upgraded my laptop HW (doubled memory, faster & double-sized SSD) and local builds have been accelerated anyway. The project itself has also been refined a litte bit. Now we have more modules to build but less dependencies and the link process is slightly faster too. Previous tests reduced build time about 43% and this configuration saves around 39%. It’s not a big difference and more or less same order on magnitude.
But the final result is that, while I work from home, I can save about 40% of my build time in average due to this build farm I have. Try it if you have a computer sitting in a drawer: it can save you a lot of time.