Bazaar

Bazaar

 




Wiki Tools

  • Find Page
  • Recent Changes
  • Page History
  • Attachments

Summary

Add support for symlinks to bzr-on-windows, so that windows users can collaborate on projects which use symlinks in their source trees.

Rationale

We want bzr to be used as a cross-platform tool, and that includes Windows developers collaborating on *nix-originated projects. These projects may include symlinks in their source trees. Windows versions prior to Vista do not support symlinks. ('Junctions' appear to support a subset of symlink functionality.) Since symlinks aren't supported, bzr currently cannot check out some projects on Windows.

Further Details

Obviously, we cannot add symlink support to Windows (even if we could, that would not fix the already released and in use versions). But it should be possible to allow Windows users to check out projects that contain symlinks, and to commit without causing those symlinks to be removed.

Assumptions

  • We want to retain symlink support

Use Cases

Project FunFoo is started on GNU/Linux. For convenience in building their code, they add some versioned symlinks to their bzr directories. Developer jrandom tries to use FunFoo, but discovers some win32-specific bugs. He gets a branch of FunFoo using bzr, fixes the bugs, commits. The *nix developers merge jrandom's fixes without issue. To them, it is not apparent that the changes were committed on Win32.

Implementation

UI Changes

When a branch is checked out on Win32, any symlinks are created as normal files. This makes the users aware of the true layout, and prevents accidentally replacing a symlink with a file. It also makes it easy to move/delete the symlink.

The file contents are:

# bzr symlink placeholder 1
This file is a placeholder for a symbolic link, because symbolic links are not supported on this platform.

(The second line may be internationalized, and is ignored by bzr.)

If the user replaces this with a file whose first line is not precisely #bzr symlink placeholder 1, that file is treated as a file. Perhaps we would add some commands to support *nix properties on Windows, e.g. bzr link, bzr set-executable, etc.

We may want to warn at branch time if the source tree contains symlinks.

We may also want to provide facilities to help users avoid creating incompatible trees in the first place.

Including the target of the link (the first argument of "ln -s" when creating the symlink in unix) in the placeholder file would be nice. Either just in the ignored section, or actually read by bzr, which would allow windows users to actually edit symlinks although they can not use them.

Code Changes

  • All symlink operations on the working tree must go through WorkingTree.

  • WorkingTree must support symlink operations (e.g. WorkingTree.readlink, WorkingTree.symlink)

  • By default, checkouts are configured to emulate symlinks if the platform does not support symlinks.
  • Even if the platform supports symlinks, emulation will be used if the checkout is configured for it. (The filesystem may be available as a network filesystem.)
  • Symlink operations are performed using the Inventory entry instead of the filesystem, if:
    • emulation is enabled
    • the InventoryEntry is a symlink

    • the on-disk filetype is "file"
    • the first line of the file is as described above

Data Migration

This change would require a bump in the checkout format, because old clients would otherwise make faulty commits.

Discussion

AlexanderBelchenko:

1. There is another variant to implement support of symlinks on pure windows. Bzr can borrow format of symlinks from cygwin. Cygwin runs on top of pure win32 file system and therefore use following workaround to store symlinks on the win32 filesystem.

Symlinks to regular files is stored in file with content:

  • '!<symlink>path/to/target\0'

This file has SYSTEM attribute (win32 file attribute) is set.

For symlinks to directories cygwin use another solution -- it use standard windows shortcuts, -- but nevertheless symlinks to directories created as described above is also recognized in cygwin as symlinks to directory.

See attached python module win32_symlinks.py that implements functions symlink and readlink to create and read cygwin symlinks on native win32. See also test_symlink and dir_symlink -- latter created by cygwin itself.

2. Based on this code I make plugin win32symlinks that provide transparent support of fake symlinks on win32: when bzr load this plugin it patch std. os, os.path modules, bzrlib.osutils modules and for bzr codes fake symlinks looks as usual symlinks. We are using Symlinks Duck Typing Mechanism. And it's even works.

3. From mailing list (AlexanderBelchenko):

***More proposal***

I think if bzr can have inside .bzr control directory additional information about subtype of working tree it will give ability to prevent bzr do illegal thing when user mix environment or try to make bzr operations with working tree from non-native platform. I have the idea to put into .bzr/checkout some additional files that will show to bzr itself on what filesystem checkout was be made and how to interpret such things as symlinks and executable bit. Probably I need to write another proposal for that things.

In short: if bzr check out tree with symlinks it can put into control directory special file, that should be valid symlink for current platform/filesystem. Because name of this file will be constant so bzr can easily detect to what filesystem this checked out working tree belongs. And if current platform name does not match to type of checkout then bzr should at least emit warnings to user about possible problems with operations that affects working tree. But better if bzr will blocks commands that operate with working tree (add, commit, status, rename/move/mv, etc.) with explanation like following (as example of concept):

"""Bzr cannot guarantee proper operations with working tree that checked out on platform XXX while it running on platform YYY. Please check out working tree of this branch on your platform or convert this working tree to native format of platform YYY."""

This approach will be backward compatible, because when in .bzr/checkout there is no special symlink file then bzr operate in usual way. Otherwise bzr will compare working tree platform with running platform and decide in what mode it should work (enable operations or block them). Also this information can be visible in output of 'bzr info' command, per example:

Format:

  • control: Meta directory format 1
  • working tree: Working tree format 3 (Cygwin) <<<<<<<<<<

    • branch: Branch format 5
    • repository: Knit repository format 1

And it possible to make conversion tool for reading symlinks produced on another platform and converting them into symlinks supported (or simulated) on current platform, because bzr has additional meta-info in inventory about what files are really symlinks. So bzr don't need to make false assumptions, and bzr will not slowdown their operations with working tree to autodetect symlinks that come in from another platform. And symlinks never be mixed between platform (from bzr's point of view).

MartinPool's answer:

This proposal looks good to me. I like the idea of leaving the symlink method unspecified until we actually need to create one so that you can move them around more easily.

We should make sure to keep the working inventory updated with information on symlinks so that we can remove or recreate them as needed.

4. From mailing list (from Nate Waddoups):

> Hello Alexander, I saw your discussion on the bzr list about using
> cygwin's pseudo-symlink method with bzr, and I wonder if you have
> considered using NTFS reparse points instead?  They are somewhat
> similar to unix symlinks.  Here is some more information about them:
>
> http://www.pcguide.com/ref/hdd/file/ntfs/filesReparse-c.html
> http://www.codeproject.com/w2k/junctionpoints.asp
>
> I am completely unfamiliar with python so I don't know if it will be
> possible to make the Win32 calls necessary to use this, but if it can
> be done, I think it might be a better solution than cygwin-style
> linking since it is supported by the OS and thus can be used by all
> Win32 applications, not just those than use cygwin's approach.
>
> I hope you find this useful,
>
> Nate Waddoups
> Redmond WA USA
> http://www.natew.com/   <== for nerds
> http://www.featherforum.com/   <== for birds

AlexanderBelchenko's answer:

Thanks for links. Probably it can be useful, but it restricted to NTFS filesystem. But very often old windows machines used FAT32. So in this case this approach cannot be used. But as alternative to cygwin-like it could be.

Googling a bit I found discussion in the python mailing list at November 2005 about NTFS reparse points. Here the root of the thread:

http://mail.python.org/pipermail/python-list/2005-November/349238.html

So I see it possible enough to implement. But unfortunately today I have not machine with NTFS and therefore I cannot implement and debug it.

If someone want to implement support of reparse points as C/C++ library (dll per example) then I could write wrapper interface to use this library from Python. But anyway support of reparse points should be independent python project, I don't see reason to include them inside bzr core.

As another solution can be used linkd.exe utility from Resource Kit (it does not ship in standard windows installation and should be installed separately):

http://support.microsoft.com/?kbid=205524

JohnMeinel's answer:

I would not recommend using reparse points. They only work directory => directory (thus you cannot do a reparse point for a file, like you can do with symlinks). They also have many more limitations (IIRC, a global limit on the number of them that can exist in a filesystem).

@JohnMeinel: That's not 100% true. Reparse points are simply special NTFS objects that are interpreted by special driver in the kernel - they can therefore be pretty much anything (say for example, a file that is actually all text files on your computer concatenated). Reparse points are used by Windows itself to implement symlinks. Now it's basically like this: Windows > 2000 and < Vista support file hardlinks and directory symlinks. Windows > Vista supports file hardlinks and both directory and file symlinks. It should further be noted that on Vista, creating a symlink is a privileged operation that requires Administrator rights, so bzr would have to require to be run as admin, or implement UAC to request admin rights specifically for the symlink operation. If it weren't for the latter, I would suggest to just require Vista and use the real thing rather than coming up with a custom application-level implementation.

@Nishi: about UAC for symlinks I found info at http://en.wikipedia.org/wiki/NTFS_symbolic_link: The default security settings in Windows Vista disallow non-elevated administrators and all non-administrators from creating symbolic links.

1. "fix security policies" - This behavior can be changed in the Local Security Policy management console (under: Security Settings\Local Policies\User Rights Assignment\Create symbolic links).

2. "UAC: run with the runas command" - It can be worked around by starting cmd.exe with Run as administrator option or the runas command.

3. "fix security policies using ntrights.exe" - Since Windows Vista Home does not have the Local Security Policy Management Console, the command line utility ntrights.exe from the Windows Server 2003 Resource Kit Tools can be used to grant rights to symbolic link creation.

So bazaar on Windows can modify security policies during install or show UAC.

To learn and play with symlinks/hardlinks/junctions you may install Link Shell Extension from http://schinagl.priv.at/nt/hardlinkshellext/hardlinkshellext.html

Again from Wiki: "Windows Vista supports a new symbolic link capability that replaces junction points in Windows 2000 and Windows XP. They are designed to aid in migration and application compatibility with UNIX operating systems." So symlinks in Vista & Win7 should be considered.

Unresolved Issues

Questions and Answers

CategorySpecification