Increasing build speed with icecc

Icecc Speed

Have you ever despaired in front of your computer waiting for a build after a minor change? Did you ever swear to all gods in that process? I did, and I usually do. Being “southern” and hot-blooded is a problem sometimes. But it also gives me the motivation to find a way to improve that.

Written by Jordi Pujol
2023/5/19

Let’s define the scenario

I’ve been working in a project with a codebase like this:

  • 2 common libraries, that can be built from time to time and installed. About 1k cpp files to build
  • Main application codebase depending on those libraries with about 3k cpp files to build 
  • Cmake based build system
  • Linux / gcc / QtCreator

The first thing I did was collecting all low hanging fruits: I used ninja as build generator, that saved me some precious time on re-building (as easy as add GNinja on the Cmake call) and I’ve activated ccache on my kit (manage kits and choose the ccache-based gcc version I wanted to use) 

But that was just a minor improvement. I needed more speed and I wanted it also when doing full rebuilds. That was impossible only with those tools, but I had a card hidden in my sleeve.

I suspect I may suffer from a slight case of hoarder syndrome, as I tend to accumulate a bunch of old computers just because I think they may be useful to me in the future. In fact, I have an old machine, that used to be my HTPC, configured to take care of my home automation control system (I installed Home Assistant there just for the fun of it).

But I still had 3 more laptops dusting around that I could use:

  • Lenovo B50-10 (Pentium N3540, @2.16GHz, 2 cores, 64 bits)
  • HP Pavilion 15 b123es (AMD Dual-Core A4-4355M, @ 1.9GHz)
  • Clonic laptop (i7-4700MQ CPU @ 2.40GHz, 12Gb memory)

The most powerful of these is my old i7 laptop. It’s now 10 y/o but yet a powerful machine. It’s usually idling around and I use it on the rare occasions I need to do some windows stuff, so it was just sitting close to me, almost useless.

So, why don’t I use it as a build server on a farm? I dug a little bit and the easiest way to test this was installing icecc, very easy to configure and set up. I did some trial and error before I got the right combination. Initially, I installed it dockerized in my windows machine, but it was unreachable from outside, so no way. Later on, I tried installing it inside WSL. I had to “hack” its auto-startup and I had to log in my user to have it active. But it caused some weird failures while building. Finally, I just made it dual-boot with a ubuntu 20.04 server install and things were smoother.

Setting up the extra CPUs

My final setup is this one:

  • XPS 15 laptop => my usual dev machine & build server (i7-6700HQ CPU @ 2.60GHz, 16Gb)
  • old i7 laptop => build server (slave)
  • Lenovo laptop => build scheduler

I only used one of the 2 less powerful laptops as scheduler. Each one was giving me only 2 extra CPUs and the overhead of managing them wasn’t worth the effort.

And here are the numbers I got. Impresive, right?

Conditions Library A Library B Main Total
icecc inactive

Items

Start time

End time

Time spent

883

13:30h

13:46h

00:16h

633

13:46h

14:00h

00:14h

3486

14:00h

15:49h

01:49h

 

 

 

02:19h

icecc active

Items

Start time

End time

Time spent

883

12:02h

12:11h

00:09h

633

12:11h

12:21h

00:10h

3486

12:21h

13:21h

01:00h

 

 

 

01:19h

Adding a second machine, with more or less same computing power, make my builds 1,76 times faster. Not a total 2x, because of network overhead and different CPU, and also because all preprocessing and linking happens in my dev machine, but good enough

In summary: I’ll keep this configuration up & running at home, as it makes me waste less time doing my daily work and caused me 0 trouble to do it. Win-win!

And here's how to do it!

If you are still reading, here comes the “how to” section.

Installing icecc was pretty straightforward in my Ubuntu machine, just a sudo apt-get install icecc icecc-monitor and you’re almost done. As you may note, I installed the monitor to check if the machines were doing good usage of the CPUs. In the other 2 machines, I just installed icecc.

Setting it up in all 3 computers was quite easy, just edit the file /etc/icecc/icecc.conf and set up this:

ICECC_NETNAME=”<choose a farm name>”

ICECC_SCHEDULER_HOST=”192.168.1.20″

In my case, I don’t even have a local DNS server, so I simply set up a fixed IP for the scheduler and wrote the same IP in all 3 machines

Of course, I set up icecc (and the scheduler, in the one that had it) to start up the service automatically with sudo service enable icecc / icecc-sceduler, this way I don’t have to think about starting it every time I power the machines on.

Don’t forget to set up this env var CCACHE_PREFIX=icecc so, you don’t have to bother about chaining icecc with ccache. It’s also the recommended way ccache maintainers want it to be done. You will avoid a lot of trouble and messages about icecc calling itself recursively.

A hint: to avoid the laptops to enter “suspend mode” when closing the lid, I had to modify the file /etc/systemd/logind.conf and change the following lines :

HandleLidSwitch=ignore
HandleLidSwitchExternalPower=ignore 

LidSwitchIgnoreInhibited=no

UPDATE: after the experiment suceeded, my next dream was buying a cheap spare datacenter server (I found for about 600€ a Dual xeon, with 24 threads available, 64Gb RAM and 4x1Tb 7.5k hot-swap sata HD in a RAID5 config) and installed that in my basement. The fun never ends!

I’ve updated my hardware, and upgraded my laptop too: I’ve duplicated the memory to 32Gb and also changed the NVMe disk to another one faster that has 1Tb capacity. Everything helps to win some speed!

The 2 old laptops have been retired in favour of this server, that hosts some virtual machines and a linux SQL server instance and stays idleing most of the time. This server acts as scheduler and as a node, as it’s powerful enough to just give me the peak I need when I’m building at home. Numbers come down here

Conditions Library A Library B Main Total
icecc inactive

Items

Start time

End time

Time spent

883

16:07h

16:12h

00:05h

644

16:12h

16:18h

00:06h

3623

16:18h

17:17h

00:59h

 

 

 

01:10h

icecc active

Items

Start time

End time

Time spent

883

17:51h

17:54h

00:03h

644

17:54h

17:57h

00:03h

3623

15:57h

18:34h

00:37h

 

 

 

00:43h

They are quite surprising: There is an improvement, but a little bit less difference than before with icecc active/inactive. In this case the reason is that I upgraded my laptop HW (doubled memory, faster & double-sized SSD) and local builds have been accelerated anyway. The project itself has also been refined a litte bit. Now we have more modules to build but less dependencies and the link process is slightly faster too. Previous tests reduced build time about 43% and this configuration saves around 39%. It’s not a big difference and more or less same order on magnitude.

But the final result is that, while I work from home, I can save about 40% of my build time in average due to this build farm I have. Try it if you have a computer sitting in a drawer: it can save you a lot of time.

Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments