Wiki code for OpenSolaris DSCM Evaluation: Mercurial

Hide Line numbers
1: = OpenSolaris DSCM Evaluation: Mercurial
2: 
3: == A. Tool Information
4: 
5: Mercurial version 0.8.
6: 
7: == B. Configuration Details
8: 
9: === 1. CPU architecture and machine specifics
10: 
11: {{{
12: 
13: kosh$ psrinfo -v
14: Status of virtual processor 0 as of: 03/30/2006 14:15:16
15:   on-line since 03/09/2006 16:44:59.
16:   The i386 processor operates at 1600 MHz,
17:         and has an i387 compatible floating point processor.
18: kosh$ uname -a
19: SunOS kosh 5.11 snv_35 i86pc i386 i86pc
20: 
21: }}}
22: 
23: === 2. Memory available
24: 
25: {{{
26: 
27: Memory size: 1024 Megabytes
28: 
29: }}}
30: 
31: === 3. Size of the repository (space)
32: 
33: 641MB, including working directory. This repository mirrors ON’s internal Teamware workspace as of 2006-04-11 16:52:19Z and has history back to build 18 (shortly after the OpenSolaris launch).
34: 
35: === 4. Number of files in the repository
36: 
37: 43,311.
38: 
39: === 5. Average number of deltas per file
40: 
41: 1.3.
42: 
43: More details: 1,884 changesets, 57,173 changes. (57173/43311=1.3)
44: 
45: == C. Evaluation Areas
46: 
47: === 1. Operation Functionality
48: 
49: (Refer to the Requirements Document for detail.)
50: 
51: ==== e1. unbiased and disconnected distribution
52: 
53: Mercurial implements a model of independent repositories, though a repository can be configured to have a [[de-facto parent>>http://www.opensolaris.org/jive/message.jspa?messageID=30643&tstart=0]].
54: 
55: Mercurial supports updates between two repositories with a common ancestor. If child-2 pulls from child-1 and has one or more changesets that aren’t in child-1 (a common scenario in ON development), child-2 will have to do an "update -m", even if the changes are non-conflicting. Unlike Teamware, the merge must then be explicitly committed. (This is a feature, since it means you can test the merge before committing it, but it seems likely to cause confusion for Sun engineers. If we use Mercurial Queues for most development work, this issue probably goes away.)
56: 
57: The "disconnected-use" requirements are all satisfied.
58: 
59: ==== e2. networked operation
60: 
61: As mentioned in the requirements document, Mercurial supports remote access via {{code}}ssh{{/code}}. Pulls can also be done over HTTP. (And of course NFS can be used for pushes and pulls).
62: 
63: ==== e3. interface stability and completeness
64: 
65: ===== storage
66: 
67: The storage representation appears to be well-documented. Certainly there’s more [[information>>http://www.selenic.com/mercurial/wiki/index.cgi/Design]] on the Mercurial website than we’ve been able to find for SCCS’s storage representation.
68: 
69: Mercurial’s storage representation does not use Unix i-numbers, so snapshots such as those provided by ZFS or Network Appliance filers should not cause problems.
70: 
71: A ZFS snapshot can be used as the source for a Mercurial clone operation.
72: 
73: At least some of the on-disk data structures do not appear to be versioned. This is a potential hazard. At least one storage representation change is planned: "RevlogNG", which is [[planned for Mercurial 0.9>>http://marc.theaimsgroup.com/?l=mercurial&m=114375669405791&w=2]]. This change is expected to address the versioning issue.
74: 
75: On-disk data structures are binary files, but I had no problems using the same repository from both SPARC and x86 systems. Binary files give improved performance, but in the unlikely event that manual repairs are needed, we’ll need a binary editing program.
76: 
77: Mercurial does not provide its own access control mechanism for controlling access to subtrees within a repository. While it might be possible to restrict user access to certain subtrees using filesystem ACLs, it would probably be better to use various pre-operation hooks (e.g., {{code}}pretxnchangegroup{{/code}}) to implement that sort of control. Bryan O’Sullivan reports that there is a recipe in the wiki for doing [[access control over {{code}}ssh{{/code}}>>http://www.selenic.com/mercurial/wiki/index.cgi/SharedSSH]], without needing to create a separate user account for everyone.
78: 
79: ===== command-line interface, hooks
80: 
81: The [[command-line>>http://www.selenic.com/mercurial/hg.1.html]] and [[hook>>http://www.selenic.com/mercurial/hgrc.5.html]] interfaces appear to be adequately documented.
82: 
83: One nit: the current documentation appears to reflect the current development version of the code, rather than the most recent release[[[1]>>#docexample]]; there is nothing in the documentation to clarify what version it applies to. If OpenSolaris uses Mercurial, we may wish to place snapshots of the code and documentation on opensolaris.org to avoid confusion.
84: 
85: The hook infrastructure invokes the named hook(s) with a few tokens such as the changeset ID passed in via the environment. This means that the hook may need to invoke various Mercurial commands to find out more about the changeset. A potential issue is that it may not always be possible to get the desired information back from the existing Mercurial command-line interfaces. For example, "hg log" gives the old and new names of a renamed file, along with the names of any other files involved in the changeset, but it can’t tell you that file "foo" was renamed to "bar". Fortunately, all the known examples of this problem are considered bugs and have fixes planned.
86: 
87: Bryan O’Sullivan reports that the Mercurial team is also considering support for Python hooks that would run in the Mercurial process.
88: 
89: Lock-reentrancy does not appear to be a problem. Mercurial does not need read locks, and hooks are currently limited to only doing read operations.
90: 
91: At least one unexpected behavior was noted while testing: pushing a changeset from repository A to repository B caused A’s {{code}}commit{{/code}} hook to fire. This appears to be a [[known bug>>http://www.selenic.com/mercurial/bts/issue113]].
92: 
93: ===== network protocol(s)
94: 
95: There is some documentation on the [[network protocol>>http://www.selenic.com/mercurial/wiki/index.cgi/WireProtocol]], though it’s a bit sketchy. The protocol is versioned.
96: 
97: ==== e4. standard operations and transactions
98: 
99: Mercurial operates on files. Rename or remove of a directory is translated into a rename or remove on the directory’s contents.
100: 
101: Rename is implemented as copy and delete. More work is needed here: merges don’t track renames, and rename conflicts are not detected.
102: 
103: Deleting a file and creating a new file with the same path is supported. The new instance includes the history of the old instance.
104: 
105: Mercurial supports the scenario where a file is deleted in one workspace, the deletion is backed out in the "gate" repository, and the file is edited concurrently in a second workspace. The backout was done via
106: 
107: >  $ hg revert -r lastrev
108:  $ hg add file
109:  $ hg commit -m "back out deletion of file"
110: 
111: where lastrev refers to the last changeset before the deletion.
112: 
113: The merge in the second workspace was a little messy: after the "update -m", the working copy of the file had the old and new versions of the file concatenated together.
114: 
115: Deleted files can still be referenced using the path and the -r option, for example
116: 
117: >  $ hg cat -r rev file
118:  $ hg diff -r rev1 -r rev2 file
119: 
120: ==== e5. per-changeset metadata
121: 
122: Mercurial associates a text comment with each changeset; this is added as part of the {{code}}commit{{/code}} operation. The first line of the comment is displayed as a summary of the changeset for operations like "log", though the full comment can be displayed with the "-v" option. This is somewhat inconsistent with the current conventions for putbacks into ON; we’ll want to think about what we want to do. Mercurial supports customization of "log" output, so that’s one option.
123: 
124: ==== c6. ease of use
125: 
126: Although I haven’t built Mercurial from source, Bryan O’Sullivan reports
127: 
128: >  It builds out of the box on Solaris 10, once the system Python’s Makefile is fixed to use gcc.
129: 
130: The primary interface is the {{code}}hg{{/code}} command, with subcommands for the various options. This is pretty standard.
131: 
132: The model is straightforward: you {{code}}commit{{/code}} one or more changesets to the repository that you’re working in, then you {{code}}push{{/code}} or {{code}}pull{{/code}} them to other repositories. Note that pushing (or pulling) a changeset updates the target repository, but updating the target’s source tree is a separate step ("hg update").
133: 
134: Mercurial offers subcommands specifically for generating and accepting source patches.
135: 
136: Mercurial supplies an HTTP server, as well. This can be used for browsing and for pulls over HTTP.
137: 
138: Support for backouts: the {{code}}revert{{/code}} subcommand can be used to back out all changesets back to a particular revision. Backing out a changeset after the files have been subsequently modified is less straightforward. There is currently an [[open feature request>>http://www.selenic.com/mercurial/bts/issue146]] for this. In the meantime, one can generate a source patch for the changeset that you want to back out, then apply the patch using "patch -R".
139: 
140: By default, "hg status" lists files that aren’t tracked in the repository (e.g., compiled binaries, editor backup files). While this is not peculiar to Mercurial, it’s a change from Teamware, and it will generate an impossible level of noise in most real-life scenarios with ON. While doing "dmake clobber" will reduce the noise considerably[[[2]>>#clobber]], that is inconvenient for a tree the size of ON. Mercurial does provide mechanisms to filter out noise (e.g., {{code}}status{{/code}} subcommand options, and {{code}}.hgignore{{/code}} files), but it’s not clear they can be used to give the desired results (show untracked source files and makefiles, but ignore all other untracked files).
141: 
142: Files should be imported with read-write permission. Mercurial keeps track of the permissions, and it complains if you try to update a read-only file (e.g., after a push or pull).
143: 
144: ===== merging
145: 
146: Mercurial’s default for resolving conflicts is the {{code}}hgmerge{{/code}} script. This script checks for the presence of various third-party conflict resolver programs, such as {{code}}tkdiff{{/code}}. At least some of these programs (e.g., {{code}}tkdiff{{/code}}) offer functionality that is comparable to what is available with Teamware and Filemerge. It should be easy to add Filemerge support to {{code}}hgmerge{{/code}} if that’s desired.
147: 
148: If {{code}}hgmerge{{/code}} can’t find any of the expected conflict resolvers, it falls back to using {{code}}diff(1){{/code}} and {{code}}patch(1){{/code}}. If {{code}}patch{{/code}} rejects any part of the diffs, {{code}}hgmerge{{/code}} invokes an editor so that the user can manually repair the conflict.
149: 
150: While first experimenting with Mercurial, I found it very easy to get my repository into a state where it would keep complaining about "outstanding uncommitted changes", but it was hard to figure out how to get out of that state. (Answer: use "hg update -C".) This probably needs to go into a FAQ.
151: 
152: ===== mismerges
153: 
154: The current version of {{code}}hgmerge{{/code}} has some problems that make it easy to introduce mismerges. One of the Mercurial developers is looking at reducing the likelihood of mismerges. We may also wish to consider our own changes.
155: 
156: First, if {{code}}hgmerge{{/code}} uses {{code}}patch{{/code}}, we may want to force a review of the changes, even if the patch applies cleanly.
157: 
158: Second, the code for invoking the editor and determining whether the conflict was resolved is a bit brittle[[[3]>>#conflicts]]. This may just be a bug that needs fixing. But we may also want a more explicit "yes I have resolved the conflicts" action from the user (which is something Subversion does).
159: 
160: There is at least one [[open issue in the Mercurial bug database>>http://www.selenic.com/mercurial/bts/issue12]] related to failed merges. Resolving this issue may address the brittleness problem mentioned above.
161: 
162: ===== intermediary snapshots
163: 
164: The current ON convention is that putbacks should not introduce SCCS deltas for intermediary snapshots or Teamware merges. This is achieved by using the {{code}}redelget{{/code}} and {{code}}reedit{{/code}} subcommands in {{code}}wx{{/code}}. Similar functionality is available with Mercurial using the [[Mercurial Queue (mq)>>http://www.selenic.com/mercurial/wiki/index.cgi/MqExtension]] extension.
165: 
166: ==== c7. no-dedicated-server mode
167: 
168: Mercurial can run without any server daemons. {{code}}ssh{{/code}} support is handled by starting a remote, transient Mercurial server automatically, which communicates with the local system over the {{code}}ssh{{/code}} connection.
169: 
170: ==== c8. tool community health
171: 
172: Mercurial has an active developer community. At least one developer (Bryan O’Sullivan) has helped with our evaluation of Mercurial, and he is interested in helping to address issues that we have run into so far (e.g., rename).
173: 
174: There have been a few Solaris-specific problems with Mercurial. The Mercurial developers have been quick to respond. And in at least one case they installed Solaris so that they could troubleshoot the issue. Also, the developers have paid attention to larger issues, rather than applying a steady stream of band-aids. For example, to address the recurring shell-compatibility problems with {{code}}hgmerge{{/code}}, one developer suggested rewriting {{code}}hgmerge{{/code}} in Python.
175: 
176: ==== c9. OpenSolaris community expertise
177: 
178: Mercurial is almost entirely written in Python; {{code}}hgmerge{{/code}} is a shell script, though there is some talk of rewriting it in Python, too. It’s unclear (to me) how much Python expertise there is in the OpenSolaris community. Fortunately, Python appears easy to learn, and the code tends to be more readable than, say, Perl. Commenting is a bit sparse, but I haven’t had problems following the code.
179: 
180: ==== c10. interface extensibility
181: 
182: Mercurial has a [[hooks mechanism>>http://www.selenic.com/mercurial/hgrc.5.html]] as well as a documented [[extensions mechanism>>http://www.selenic.com/mercurial/wiki/index.cgi/ExtensionHowto]]. Some hooks can abort the current operation.
183: 
184: ==== c11. transactional operations and corruption recovery
185: 
186: Mercurial’s state files are updating by appending. So corrupted files can be repaired by rolling back to a consistent set of files.
187: 
188: Signal handling (e.g., {{code}}SIGINT{{/code}}) appears sensible.
189: 
190: I simulated a crash using {{code}}SIGKILL{{/code}} during a clone operation and was able to get the workspace into a state where there were many missing files. That is, the files were in the repository, but they were not in the visible source tree. This experiment raised a couple issues that we’ll want to look at more carefully.
191: 
192: * the child repository was left with two lock files which would block some operations but not others. A push to the parent was not blocked. Either this is a bug or we need to understand Mercurial’s use of locks better.
193: * Mercurial seemed confused by the contents of the child. I edited a file in the workspace~--{{code}}README.opensolaris{{/code}}~--and tried to commit it, but Mercurial said there were no changes. Mercurial let me clone the child, edit {{code}}README.opensolaris{{/code}} in the second child, and push the changes to the first child. Trying to work with {{code}}README.opensolaris{{/code}} in the first child was still troublesome~--it was as though Mercurial did not realize that {{code}}README.opensolaris{{/code}} was already in the repository. For operational simplicitly, we’ll want some sort of cleanup and repair script, with simple guidelines on when to run it.
194: 
195: There also appear to be a couple open issues related to locking[[[4]>>#locking]].
196: 
197: ==== c12. content generality
198: 
199: Mercurial supports binary files as well as text. However, the merge code appears to assume text files. We’ll need to think about to handle binary files.
200: 
201: ==== o13. partial trees
202: 
203: Not currently supported, though there have been discussions about adding support for it..
204: 
205: ==== o14. per-file histories
206: 
207: Changesets are for the entire repository, not per-file. But you can get a per-file history by specifying the file name with "hg log".
208: 
209: === 2. Storage
210: 
211: We didn’t have any problems running out of storage or swap. No storage spikes were observed.
212: 
213: One thing that deserves further investigation is the storage consumed by the conflict tests in the test harness that we used. The first test introduces a content conflict in usr/src/cmd/sort/Makefile.com. The second test introduces a couple rename conflicts in usr/src/cmd/pwd. This led to a size increase of 2.3MB in the test repository, which seems excessive.
214: 
215: === 3. Performance
216: 
217: A local clone of the OpenSolaris ON 20060222 tree takes a couple minutes on the above hardware using ZFS; about twice that on UFS.
218: 
219: Using the [[repo with history>>#repo]], performance looks like this (times are in mm:ss):
220: 
221: |=operation|=Local (zfs)|=LAN (NFS)|=LAN (ssh)|=WAN (Menlo Park-United Kingdom) (ssh)
222: |clone (6 samples)|2:55 (std dev: 1:00)|6:54 (std dev: 0:17)|4:01 (std dev: 0:08)|35:42 (std dev: 2:45)
223: |local commit (1 file; 6 samples)|0:23 (std dev: 0:13)|0:17 (std dev: 0:09)|0:19 (std dev: 0:11)|0:19 (std dev: 0:10)
224: |push (1 file, 1-3 deltas; 4 samples)|0:01|0:02|0:03|0:06
225: |pull (1 file, 1-3 deltas; 6 samples)|0:01|0:01|0:02|0:06
226: 
227: == D. Changes/Features Required/Desired
228: 
229: === Must Have Initially
230: 
231: * rename/merge integration
232: * versioning of on-disk data structures (this should be in the upcoming RevlogNG changes)
233: * a reliable mechanism for checking a tree (repository plus working directory) after a crash and making any necessary repairs (see [[Section c11>>#c11]] above)
234: 
235: === Want Eventually
236: 
237: * detection of rename conflicts
238: * supported interface for showing renames and deletions in the log (planned for Mercurial 0.9)
239: * better handling of failed merges ([[issue12>>http://www.selenic.com/mercurial/bts/issue12]] in the Mercurial issue database)
240: 
241: == Notes
242: 
243: [1] For example, the 2006-03-22 version of the {{code}}hgrc(5){{/code}} man page lists hooks that were apparently added after 0.8, although 0.8 is the most recent release.
244: 
245: [2] Besides leaving editor backup files, our clobber builds leave some generated files behind.
246: 
247: [3] The relevant code is
248: 
249: {{{
250: 
251: $EDITOR "$LOCAL" "$LOCAL.rej" && test -s "$LOCAL.rej" || exit 0
252: 
253: }}}
254: 
255: If {{code}}$EDITOR{{/code}} can’t be invoked for some reason, we’ll take the "exit 0", which indicates a successful merge.
256: 
257: [4] [[issue132>>http://www.selenic.com/mercurial/bts/issue132]] "hg should revalidate its data after locking the repo" and [[issue154>>http://www.selenic.com/mercurial/bts/issue154]] "race between undo and all readers"
258: 
259: == History
260: 
261: ; 2006-04-20
262: : Resolved TBDs. More information on use with ZFS snapshots. Minor editorial changes.
263: ; 2006-04-13
264: : Updates after email discussion with Bryan O’Sullivan <bos@serpentine.com>
265: ; 2006-03-31
266: : First draft.
267: 
268: // Last change: 2006-04-20 11:29 PDT //
OpenSolaris

Top Menu

Wiki code for OpenSolaris DSCM Evaluation: Mercurial

Search

Collectives

Community Group

Project

User Group

Subsites

Community Group tools Pages
