Related page: RoadmapForHPSS.

Overview

This page describes work to improve Bazaar network performance over the bzr://, bzr+ssh:// and bzr+http:// network protocols.

We test several representative scenarios, using a simulated network connection with performance similar to a Brazil-London link. We measure the total time for the scenario and also the breakdown across individual calls. We then analyze which of these operations can be eliminated altogether or combined into more efficient methods. For each scenario we also have some data on performance in previous revisions and a goal time.

Method

Same as SmartPushAnalysis1.4, except using bzr.dev from mid-September 2008 rather than 1.4.

Key observations

• Opening a bzrdir, branch and repository requires 7 round-trips (and thus 7s). Ideally this should take just one.
• Creating a bzrdir, branch and repository is extremely wasteful. Approximately 30s spent to create the bzrdir and repository, and approximately 40s to create the branch. Ideally this too should take just one round trip.

• Before bzr can push new revisions, it first has to determine which revisions need to be pushed. This is done by _walk_to_common_revisions, and currently happens over VFS, visible as readv RPCs to *.rix files. The equivalent get_parent_map calls would be much faster; see Note 1 in Case 3's analysis.

• Checking that parents exist for new file text deltas takes a significant fraction of time for incremental pushes.

Tests from SmartPushAnalysis1.4

The test cases are the same as used for SmartPushAnalysis1.4.

Case 1: Push of new standalone branch (format 5 branch, pack-0.92 repo)

Bzr 1.4 time: 609s
bzr.dev Sept '08 time: 441s

XXX

Case 2: Push 1 new revision onto existing standalone branch (format 5)

Bzr 1.4 time: 97s
bzr.dev Sept '08 time: 67s

XXX

Case 3: Push 1 new revision onto existing standalone branch (format 6)

Bzr 1.4 time: 71s
bzr.dev Sept '08 time: 61s

RPC summary

• 0-6s: opening the branch
• 6s: lock branch
• 7s: get last_revision_info
• 8-11s: VFS open repository
• 'Using fetch logic' log message
• 11-22s: readvs of the single rix index

  • (see note 1 in the Analysis below)

• 22-25s: put (0 bytes) + append (42 bytes, just the file header) of new pack file to upload area
• 25-37s: readvs of the single tix index (doing _check_references?)
• 37s: one more append to new pack file
• 38-42s: put new rix, iix, tix, six
• 42s: rename new pack file into place
• 43-47s: VFS lock repository
• 48s: put new pack-names file
• 49-53s: VFS unlock repository
• 53s: single short readv of new pack file
• 54s: set_last_revision
• 55-59s: VFS reads including: branch format, repo format, lock info file

  • (see note 2 in the Analysis below)

• 60s: Branch.unlock

Analysis

Note 1: The extensive rix readvs are due graph queries by InterPackRepo, which InterOtherToRemote delegates to. InterPackRepo uses the VFS-backed repository objects, so the better get_parent_map RPC is never used.

I have an experimental branch that lets InterOtherToRemote override how the InterRepo it delegates to does graph queries, i.e. that makes this case use the smart get_parent_map RPC instead. It also improves the get_parent_map RPC client to cache negative results too. The whole sequence of rix reads is then replaced with just two get_parent_map calls (although a single short rix readv is incurred later on). Net win: 10s.

A further improvement is to change InterRepo._walk_to_common_revisions to query for many parents at a time (say 50?), rather than one at a time. This cuts the two get_parent_map calls down to just one, and for pushes of many revisions the benefit should be much greater.

Note 2: These VFS reads are probably due to the self.tags.merge_to(...) call in _basic_push. That call is unnecessary for the test branch, which has no tags. There is a patch on the list already that skips this step in this case.

Case 4: Push a diverged branch, resulting in an error

Bzr 1.4 time: 55s
bzr.dev Sept '08 time: 44s

RPC Summary

• 0-6s: opening the branch
• 6s: lock branch
• 7s: get last_revision_info
• 8-11s: VFS open repository
• 11-22s: readvs of all two rix indices.
  • See comments about get_parent_map above; a net win of 3s is achieved by the same change mentioned there.
• 23s: put + append of new pack file

• Note: no tix readvs occur, unlike case 3, presumably because there are no new file texts.

• 25-30s: put of new rix, iix, tix, six
• 30-31: rename pack into place
• 31-35s: VFS lock repository
• 35-37s: put new pack-names file
• 37-41s: VFS unlock repository
• 41s: single short readv of new pack file
• 42s: set_last_revision_ex call
• 43s: Branch.unlock

Analysis

Exactly the same events as case 3, except the new revision in this test has no new file texts (it was generated with bzr ci --unchanged), so no tix readvs occur. In practice I think normally there will be new file texts and thus be the same as case 3.

It makes sense that this case is as costly as case 3: it has to do the same uploading of the new data, and the only difference is that the penultimate call, set_last_revision_ex, fails. It's hard to fail sooner, as neither side knows that the two tips are divergent until they both exist in the one repository. The final Branch.unlock is necessary in both cases.

Case 5: Push a diverged branch using --overwrite

Bzr 1.4 time: 122s
bzr.dev Sept '08 time: 51s

RPC Summary

Identical to case 4 up to 42s:

• 42-44s: set_last_revision call
• 44-48s: VFS reads including: branch format, repo format, lock info file
• 49s: Branch.last_revision_info
• 50-51s: Branch.unlock

Analysis

Again, the VFS reads and last_revision_info call here are probably due to the redudant self.tags.merge_to(...) call. See note 2 in the analysis of case 3.

Case 6: Push new branch into shared repo, only 1 new revision

Bzr 1.4 time: 125s
bzr.dev Sept '08 time: 124s (a 1s change is within the margin of error)

• 0-23s: create and open bzrdir
  • Two calls to mkdir('/devel') at 2s and 3s. The second one fails, obviously.

  • total of four useless calls made: the doomed mkdir, and three failed gets of files that can't possibly exist (because we just made that directory).

• 23-25s: BzrDir.find_repositoryV2, Repository.is_shared

• 25-28s: vfs gets and stats of other basic repo files and directories.

• 28-39s: Many readvs of rix files. See Note 1 of Case 3's analysis

• 39s: stat /.bzr
• 40-42s: create new pack file, write its header

• 42-55s: Many readvs of tix files.

• 55-56s: write 3474 bytes to new pack file.
• 56-60s: upload rix, iix, tix, six files.
• 60s: rename new pack file to final location

• 61-71s: vfs lock repo, get & put pack-files, vfs unlock repo

• 71s: single rix readv of new pack file
• 72-89s: vfs create branch
• 90-94s: re-open bzrdir and repo
• 94s: Repository.lock_write
• 95-106s: vfs lock branch, put last-revision, vfs unlock branch

• 107-117s: vfs lock branch, get & put branch.conf, vfs unlock branch

• 117-124s: re-read/re-open bzrdir, branch, repo

  • vfs has no-working-trees

  • vfs get branch, repo, bzrdir format files

  • BzrDir.open, BzrDir.find_repositoryV2, Repository.is_shared

Analysis

BzrDir and Branch creation is very wasteful.

This really ought to be much faster: the new revision is only 3474 bytes!

Case 7: Push new branch into empty shared repo

Bzr 1.4 time: 571s
bzr.dev Sept '08 time: 434s

RPC Summary

• 0-31s: create and open bzrdir, repository.
  • Two calls to mkdir('/devel') at 2s and 3s. The second one fails, obviously.

  • total of four useless calls made: the doomed mkdir, and three failed gets of files that can't possibly exist (because we just made that directory).

• 31-345s: upload data to a new pack file in approx. 1MB chunks.
• 346-369s: upload one each of rix, iix, tix, six files.
• 369s: move pack file into place

• 370s-380s: vfs lock repo, get & put pack-names, vfs unlock repo

• 380-382: single rix readv.
• 382-400s: vfs create branch.
• 400s: try to open a repository co-located with the new branch (even though there can't possibly be one there)

• 401-404s: BzrDir.open, BzrDir.find_repositoryV2, Repository.is_shared

• 404s: Repository.lock_write
• 405-416s: vfs lock branch, put last-revision file, vfs unlock branch
• 416s: Repository.lock_write (why? And why does it succeed if the repo is already locked?!)

• 417s-427s: vfs lock branch, get & put branch.conf, vfs unlock branch

• 427s: vfs has no-working-trees

• 428s-430s: vfs get branch, repo, bzrdir format files

• 431-433s: BzrDir.open, BzrDir.find_repositoryV2, Repository.is_shared

Analysis

Almost everything outside of the "31-345s: upload data to new pack file" is waste. i.e. transferring the bytes takes about 310s (+ 25s for the bytes of the index files), so there's at least 100s out of 434s of wasted time here.

30s to fail to open and then create a bzrdir and repository is pretty ridiculous. In principle this could be done in a single round trip! When you factor in branch creation too it's even worse.

New Tests

Case 8: push 0 new revisions

bzr.dev & 1.7 time: 26s
faster-empty-push branch time: 8s (this improvement should land in 1.8)