Mercurial vs. Bazaar speedtest clone and log - update: hg 1.1 vs. bzr 1.10

I repeatet my test [1] with the provided Python 2.x repos from the DVCS PEP [2] for Python to check the performance of Bazaar and Mercurial.

All these tests are done only once with some mostly constant load, so they don't qualify as scientific tests, but they give a good impression of the differences between Bazaar (bzr) and Mercurial (hg).

Versions:
- Bazaar 1.10
- Mercurial 1.1

This comparision should be fair since Bazaar 1.10 is more recent, but Mercurial 1.1 is a major release.

You can do all these tests yourself using the script from the hg-vs-bzr-unscientific repository:
hg clone http://www.bitbucket.org/ArneBab/hg-vs-bzr-speedtest-unscientific/ [3] cd hg-vs-bzr-unscientific ./test.sh

Results

First test: Initial cloning from the web.

With repositories containing the same changesets (roughly, since bzr tracks dir name changes and hg doesn't) Mercurial is more than 6 times as fast as Bazaar. But there might be a better way to serve bzr repositories, so this isn't perfectly decisive. But since local cloning in Mercurial is 6.7 times faster, the reason for the higher speed is likely to be in the client.

Second test: Repository size

A Mercurial repository needs about half the space of a Bazaar repository, even though they contain roughly the same data. Manual repacking in Bazaar doesn't change this size significantly (213M instead of 214M).

Third test: A full log.

Here Mercurial is about 2.2 times as fast as Bazaar: 13s instead of 30s.

Fourth test: Local clones

Local cloning in Mercurial is about 6.7 times faster than in Bazaar: about 38s instead of 4min 14s.

Fifth test: Local clones with shared repository and forced hardlinks (bzr)

As suggested in the Mercurial Mailinglist, I reran the local cloning test for Bazaar using a shared repository and forced hardlinks. With this Bazaar needed just 4.5s for a local clone, without hardlinks about 12s, so by using shared repositories the local cloning becomes quite fast in Bazaar (as long as you restrict yourself to the shared repository).

Test	Mercurial	Bazaar	Result
Initial clone	1m23.699s	8m52.962s	Mercurial is about 6 times faster
Repository size	111M	214M	Mercurial needs roughly half the space
full log	0m12.907s	0m29.207s	Mercurial is about 2.2 times faster
local clone	00m37.642s	4m14.102s	Mercurial is about 6.7 times faster
local clone with hot buffers	0m32.744s	3m44.461	Mercurial is about 6.8 times faster
local clone with hot buffers and shared repository and hardlinks (bzr)	0m32.744s	0m4.546s	Bazaar is about 7.2 times faster (but you must create your clones in the shared repository)

Data

This is a clean run of the script plus two reruns of the last test: hot copy of .bzr and .hg, since there was an error in the script.

$ ./test.sh bzr --version No handlers could be found for logger "bzr" Bazaar (bzr) 1.10 Python interpreter: /usr/bin/python 2.5.2 Python standard library: /usr/lib64/python2.5 bzrlib: /usr/lib64/python2.5/site-packages/bzrlib Bazaar configuration: /home/arne/.bazaar Bazaar log file: /home/arne/.bzr.log

bzr comes with ABSOLUTELY NO WARRANTY. bzr is free software, and
you may use, modify and redistribute it under the terms of the GNU
General Public License version 2 or later.

hg --version
Mercurial Distributed SCM (version 1.1)

Copyright (C) 2005-2008 Matt Mackall and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

initial clone bzr
time bzr branch http://code.python.org/python/trunk [5] python-bzr-trunk
No handlers could be found for logger "bzr"
[18778] 2008-12-13 12:39:35.629 INFO: Branched 40761 revision(s).
Branched 40761 revision(s).

real 8m52.962s
user 5m52.733s
sys 0m10.519s

initial clone hg
time hg clone http://code.python.org/hg/trunk/ [6] python-hg-trunk
requesting all changes
adding changesets
adding manifests
adding file changes
added 40690 changesets with 86495 changes to 8167 files
updating working directory
3921 files updated, 0 files merged, 0 files removed, 0 files unresolved

real 1m23.699s
user 1m0.614s
sys 0m9.797s

comparing repository sizes
du -hs python-bzr-trunk/.bzr python-hg-trunk/.hg
214M python-bzr-trunk/.bzr
111M python-hg-trunk/.hg

time for a full bzr log
time bzr log >/dev/null
No handlers could be found for logger "bzr"

real 0m29.207s
user 0m28.022s
sys 0m0.378s

time for a full hg log
time hg log >/dev/null

real 0m12.907s
user 0m12.385s
sys 0m0.244s

bzr local clone
time bzr branch python-bzr-trunk/ python-bzr-trunk2
No handlers could be found for logger "bzr"
| [=============================================== ] Copying content texts 3/5^[18796] 2008-12-13 12:45:55.641 INFO: Branched 40761 revision(s).
Branched 40761 revision(s).

real 4m14.102s
user 3m23.507s
sys 0m6.860s

hg local clone
time hg clone python-hg-trunk python-hg-trunk2
updating working directory
3921 files updated, 0 files merged, 0 files removed, 0 files unresolved

real 0m37.642s
user 0m21.634s
sys 0m5.216s

clone with hot filesystem buffers, hg
rm -r python-hg-trunk
time hg clone python-hg-trunk2 python-hg-trunk
updating working directory
3921 files updated, 0 files merged, 0 files removed, 0 files unresolved

real 0m32.744s
user 0m21.820s
sys 0m4.431s

clone with hot filesystem buffers, bzr
rm -r python-bzr-trunk
bzr branch python-bzr-trunk2 python-bzr-trunk
No handlers could be found for logger "bzr"
[18802] 2008-12-13 12:51:30.879 INFO: Branched 40761 revision(s).
Branched 40761 revision(s).
rm -r python-bzr-trunk2
time bzr branch python-bzr-trunk python-bzr-trunk2
No handlers could be found for logger "bzr"
[18816] 2008-12-13 12:55:16.111 INFO: Branched 40761 revision(s).
Branched 40761 revision(s).

real 3m44.461s
user 3m22.856s
sys 0m5.401s

hot copy of .bzr
cp -r python-bzr-trunk/.bzr /tmp/tmp-bzr
rm -r tmp-bzr
rm: Entfernen von „tmp-bzr“ nicht möglich: Datei oder Verzeichnis nicht gefunden
time cp -r python-bzr-trunk/.bzr /tmp/tmp-bzr

real 0m2.979s
user 0m0.012s
sys 0m1.271s

hot copy of .hg
cp -r python-hg-trunk/.hg /tmp/tmp-hg
rm -r tmp-hg
rm: Entfernen von „tmp-hg“ nicht möglich: Datei oder Verzeichnis nicht gefunden
time cp -r python-hg-trunk/.hg /tmp/tmp-hg

real 0m8.213s
user 0m0.079s
sys 0m2.476s
rm -r python-bzr-trunk python-bzr-trunk2 python-hg-trunk python-hg-trunk2 /tmp/tmp-hg /tmp/tmp-bzr

# testing the hot copy of .bzr and .hg again, but this time really deleting /tmp/tmp-*

$ rm -r /tmp/tmp-* ; time cp -r python-hg-trunk/.hg /tmp/tmp-hg ; time cp -r python-bzr-trunk/.bzr /tmp/tmp-bzr ;

real 0m3.183s
user 0m0.079s
sys 0m2.250s

real 0m2.139s
user 0m0.014s
sys 0m1.593s

$ rm -r /tmp/tmp-* ; time cp -r python-hg-trunk/.hg /tmp/tmp-hg ; time cp -r python-bzr-trunk/.bzr /tmp/tmp-bzr ;

real 0m3.921s
user 0m0.059s
sys 0m2.691s

real 0m3.350s
user 0m0.013s
sys 0m1.632s

An additional test with a bzr optimized workflow gives the following results:

bzr optimized local cloning test using a shared repo. time bzr init-repo bzr No handlers could be found for logger "bzr" Shared repository with trees (format: rich-root-pack) Location: shared repository: bzr

real 0m0.442s
user 0m0.328s
sys 0m0.069s
cd bzr
time bzr branch http://code.python.org/python/trunk [5] python-bzr-trunk
No handlers could be found for logger "bzr"
[26234] 2008-12-15 11:55:14.502 INFO: Branched 40773 revision(s).
Branched 40773 revision(s).

real 10m39.557s
user 5m28.317s
sys 0m9.968s

check the size after repacking
bzr pack
No handlers could be found for logger "bzr"
rm .bzr/repositry/obsolete_packs/*
rm: Entfernen von „.bzr/repositry/obsolete_packs/*“ nicht möglich: Datei oder Verzeichnis nicht gefunden
du -hs .bzr
213M .bzr
time bzr branch python-bzr-trunk python-bzr-trunk2 --hardlink
No handlers could be found for logger "bzr"
[26879] 2008-12-15 13:07:07.532 INFO: Branched 40773 revision(s).
Branched 40773 revision(s).

real 0m4.546s
user 0m3.682s
sys 0m0.737s
cd ..
rm -r python-* /tmp/tmp-hg /tmp/tmp-bzr bzr/

Notes

To alleviate some doubts about system load, I stopped all other operations besides this test, konqueror and kmail. During the inital bzr clone top gave me the following (on my system everythign I start from an X session has niceness 1 so X doesn't bog down core processes):
top - 12:31:07 up 1 day, 2:36, 5 users, load average: 0.70, 0.99, 0.95 Tasks: 146 total, 3 running, 143 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 2.0%sy, 97.7%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st Mem: 2060384k total, 1146140k used, 914244k free, 103944k buffers Swap: 5111796k total, 4048k used, 5107748k free, 112700k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18778 arne 21 1 209m 89m 3960 R 94.7 4.4 0:14.24 bzr
15922 arne 21 1 416m 134m 42m S 3.3 6.7 4:40.40 konqueror
7591 root 21 1 651m 310m 6568 S 1.0 15.4 21:54.84 X

Bazaar once managed to get a local clone in a shared repository with forced hardlinks done in just 2.598 seconds, but every subsequent run gave results in the range of 4.3s to 4.8s, so that speed wasn't repeatable and I used a medium value for evaluation (about 4.5s).

This definitely shows the danger of unscientific tests: You don't know how unsure the results are - the difference in this case was about 50%, with the variance yet to be determined.

Result

So Mercurial is far faster than bzr when it comes to cloning (except when using bzr with a shared repository), especially for the initial clone, and a good deal faster in generating the log, while being a lot more newbie friendly than git (especially when the newbies are former svn users - Python currently uses svn).

Update: Python decided to switch to Mercurial [7].

Mercurial vs. Bazaar speedtest clone and log - update: 4 runs with different versions

Some folks in #mercurial @ freenode.net just repeated the tests, so we have now a bit more stable data [8].

The evaluation shows the following:

Initial clone: hg is about 4.4 times faster (about 2 min vs. 6 to 15 min)
Repository sizes: the hg repo is about 1.92 smaller (~113M vs. 215M)
Time for a full log: hg is about 2.36 times faster (~21s vs. ~50s)
Time for annotating Misc/NEWS: hg is 1.5 times slower than bzr.
Without the result from bzr-1.6.1 it is 2.6 times slower (~43s vs 17s).
Integrity checking: hg is by several orders of magnitude faster than bzr which just took too long - everyone stopped it after varying time (30s to 17 min), because the output spoke of hours remaining, one had an integrity error. hg needed about 1 min.
Local clone: hg is 11 times faster (39s vs. 7.14 min).
Without the 1m15 result from the high disk load host it is 16 times faster (26s).
Local clone with hot filesystem: hg is 14.9 times faster (26s vs. 6.5 min).
Hot copy of just .bzr / .hg: The speeds are about equal, so the difference doesn't come from raw filesystem speed (2s).
Additional Bazaar tests to check shared repository cloning performance (you only get this when you use a shared repository and only clone that shared repository): With shared repository and hardlinks bzr only needs about 5 seconds for cloning.

So Mercurial is faster than Bazaar for the initial cloning and for free cloning (Bazaar takes almost 10 mintes for the initial clone while Mercurial finishes after just 2 minutes), while Bazaar is faster when cloning in a shared repository.

Also Mercurial is more than 2 times as fast as Bazaar in generating the log, while Bazaar is faster at annotating.

Additionally Mercurial consumes about half as much diskspace as Bazaar.

The biggest difference is in integrity checking, though, where all testers stopped Bazaar after up to 17min, since it didn't even check a fraction of the repository in that time. Mercurial on the other hand finished after just one minute.