(SV*)null
interpreting the void as SamV
People who are responsible for Subversion servers and are converting them to git in order to lay them down to die are advised to consider the one-off git-svnimport, which is useful for bespoke conversions where you don't necessarily want to leave SVN/CVS/etc breadcrumbs behind.
There are lots of options here. We can just check out the head, we can do a full import from the master server, or we can convert our existing SVK mirror paths.
You can check out the entire history of the project with:
$ git-clone git://utsl.gen.nz/parrot Initialized empty Git repository in /home/samv/tmp/parrot/.git/ remote: Generating pack... remote: Done counting 152636 objects. remote: Deltifying 152636 objects. remote: 100% (152636/152636) done Indexing 152636 objects. remote: Total 152636, written 152636 (delta 102789), reused 152478 (delta 102789) 100% (152636/152636) done Resolving 102789 deltas. 100% (102789/102789) done Checking files out... 100% (2990/2990) done
Great! Didn't take long, and that was the same as the whole svk co sequence - add mirror, sync revisions, and checkout. If you are close enough on the network to utsl.gen.nz, that may have taken less than a minute. You can proceed to using your git-svn git repository, below, if you just want to play with it and not worry about the painful migration part.
Of course if svn.perl.org is down and you don't have an SVK mirror then this is your only option.
Though you don't need it to follow most of this tutorial, a version of git-core more recent than the one provided by your distribution is almost certainly going to give you fewer issues in the long run.
Get yourself a tarball release of git - repo.or.cz is probably a good place to go. git itself is fairly simple to build (apart from the docs, which have a dependency called asciidoc - but just read the .txt files under Documentation/ in lieu of getting that if you have difficulty). You'll also need to install the Subversion SWIG bindings to get git-svn to work, which of course is a world of pain that I won't go into here. You'll also want Tk installed for one of the most important GUIs.
It is probably second fastest to just check out the SVN head using git-svn; this is a bit like setting up a mirror path with svk mirror, then syncing only to the head revision using svk sync -s NNN (where NNN is the head revision, found below using svn log):
$ svn log https://svn.perl.org/parrot/trunk|head ------------------------------------------------------------------------ r17048 | bernhard | 2007-02-19 07:32:13 +1300 (Mon, 19 Feb 2007) | 3 linesRemove the PIR.pg and bc.pg examples as they are now covered by languages/abc and languages/PIR.
------------------------------------------------------------------------ r17047 | bernhard | 2007-02-19 07:09:00 +1300 (Mon, 19 Feb 2007) | 5 lines
[languages/PIR] $ mkdir parrot $ cd parrot $ git-svn init https://svn.perl.org/parrot/trunk Initialized empty Git repository in .git/ git-svn Using higher level of URL: https://svn.perl.org/parrot/trunk => https://svn.perl.org/parrot $ git-svn fetch -r17048 A DEPRECATED.pod A debian/libparrot-dev.install A debian/parrot-doc.install ... A examples/streams/ParrotIO.pir A examples/streams/Include.pir A examples/streams/Filter.pir r17048 = a57c09abef48d73f3c74c6a307793301b5956bfd (git-svn) Checking files out... 100% (2959/2959) done Checked out HEAD: https://svn.perl.org/parrot/trunk r17048
$
Well, that was almost as quick - under 2 minutes for a head checkout; it had to download about as much as a release tarball. If you like, from here you can proceed to using your git-svn git repository.
But people who use git are used to treating their repositories as a revision data warehouse which they use to mine useful information when they are trying to understand a codebase.
We can't do that, but once your git-fu is strong, you can see it is easy to graft on the earlier history if you want to, using history rewriting. I'll briefly mention grafting (and its drawbacks) later on.
So, it is better to have the complete project history converted, but you probably won't want to wait the day or two it can take to replay a moderately sized Subversion repository using SVK (can anyone mirror the 48GB KDE Subversion repository?).
The support for this isn't yet in a released git, so until it gets merged you'll need to clone git://git.bogomips.org/git-svn.git and build and install that. Look for --useSvmProps in git-svn init -h to see if your git-svn is new enough..
First, svk mi -l will tell us where the mirror paths are.
$ svk mi -l | grep parrot /parrot/master https://svn.perl.org/parrot $
That's everything we need to get started. Now we just need to convert /parrot/master to an SVN url; the depot is everything up to the second "/", and most SVK users will just be using a single depot with an empty name, //
$ svk depotmap -l | grep '/parrot/' /parrot/ /home/samv/.svk/parrot $
So, I take the depot path and add on the rest of the mirror path, I should be able to look at the path using plain svn;
$ svn pl file:///home/samv/.svk/parrot/master Properties on 'file:///home/samv/.svk/parrot/master': svm:source svm:uuid svk:merge $ svn ls file:///home/samv/.svk/parrot/master branches/ tags/ trunk/ $
Great! The pl (proplist) command was important - the properties there, particularly svm:source and svm::uuid, must be there for git-svn to convert this repository correctly. We use the --useSvmProps option to set up the repository rewriting:
Set up the fetch using git-svn init:
$ git-svn init -t tags -b branches -T trunk \
--useSvmProps file:///home/samv/.svk/parrot/master
Initialized empty Git repository in .git/
Using higher level of URL: file:///home/samv/.svk/parrot/master => file:///home/samv/.svk/parrot
$
git-svn is quite capable of tracking multiple Subversion repositories that hold mirrors of the same project, though of course probably most people actually doing that are SVK users, and the "other repository" is your local depot. The above command set up a git-svn remote with the default name of "svn". Take a look at what was configured by running cat .git/config.
$ cat .git/config
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[svn-remote "svn"]
url = file:///home/samv/.svk/parrot
fetch = trunk:refs/remotes/trunk
branches = branches/*:refs/remotes/*
tags = tags/*:refs/remotes/tags/*
$
All look good? So,
$ git-svn fetch --repack 1000 --useSvmProps
A README
r2 = 5c2dbc76df3fc7569d0b779841427d5ddf406e9d (trunk)
M README
r3 = 9aa2f03a26ed9617cf7002bbe4acae5d3d24dadf (trunk)
...
$
So once that's all complete what did we win so far?
$ du -sk //home/samv/.svk/parrot .git 353576 //home/samv/.svk/parrot 155245 .git $
Well, that's a bit of savings. git saved half the space compared to Subversion fsfs. But it turns out that a lot of it is just git-svn metadata. And we can compress it more; I've got CPU to burn so I ran this command:
$ git-repack -a -d -f --window 100 Generating pack... Done counting 131402 objects. Deltifying 131402 objects. 100% (131402/131402) done Writing 131402 objects. 100% (131402/131402) done Total 131402 (delta 99440), reused 31385 (delta 0) Pack pack-079a95f55810fc1eea600bc89c911a2bf85c1add created. $ ls -l .git/objects/pack/ total 33745 -r--r--r-- 1 samv samv 3154712 2007-02-20 16:00 pack-079a95f55810fc1eea600bc89c911a2bf85c1add.idx -r--r--r-- 1 samv samv 31360284 2007-02-20 16:00 pack-079a95f55810fc1eea600bc89c911a2bf85c1add.pack $
You may be wondering, "353MB of Subversion repository squeezed into 31MB of git pack? That's smaller than an SVN head checkout! Have not all the revisions been copied? Did something get missed?"
It turns out that git is just being incredibly space-efficient. More incredible stories about shrunken repositories can be found all over the internet. Talk to the GCC, Mozilla and KDE folk for the most impressive ones.
Now, in theory, we could keep using SVK to mirror revisions, and keep using git-svn fetch to copy them into the git repository. But we want some more space on our laptop to hold more MP3s, so we'll eventually delete it. Ideally we also want to convert our local branches - getting this working is still on my TODO list, but the intention is that git-svn will be extended to perform this functionality.
This procedure is the same as the SVK one above, but we can just use the published repository URL.
$ mkdir parrot $ cd parrot $ git-svn init -t tags -b branches -T trunk https://svn.perl.org/parrot Initialized empty Git repository in .git/ $ git-svn fetch ...
I didn't test this one - I have already waited the many hours it took to sync the first time. Doing this for FAI took days. And the repository had the sheer indencency to end up tiny.
What I'll do now is go through the SVK tutorials and convert them to git-svn, then introduce some examples of stuff that you can do with git that is difficult to get right using SVK. Actually why not get onto some cool stuff first.
This is your all-seeing eye. You can crank open gitk on it and click on commits, see their patches and the state of the tree at that point in time.
$ gitk --all
gitk does some really cool things but is most useful when looking at projects that have cottoned onto feature branches (see feature branches, below). If you're looking at a project where everyone commits largely unrelated changes to one branch it just ends up a straight line, and not very interesting.
So far we've got as far as the equivalent of svk mirror and svk sync. Didn't we miss svk depotmap?
Git normally stores its repository information under .git at the top level of your checkout. But everything's compressed and the filenames don't resemble the files in your checkout so grep -r and find etc don't hate you. You can set GIT_DIR to get all the tools to look somewhere else if you really care, but for most people this system works very well. GIT_DIR doesn't work the same as SVKROOT in SVK, it's a per-checkout path, not pointing to a central place.
I don't know about you but I was always running into situations where my ~/.svk/config didn't match reality, and there were no breadcrumbs left in the checkout to do anything with it. I much prefer these floating repositories and I hear that they have been recently added to SVK.
One of the nice things about git (and darcs and bzr and ...) is that to make branches is simple. Say you want to take a directory, and work on it somewhere else in a different direction, you can just make a copy. Contrast this with Subversion, where you have to do some crap with the branches/ paths and svn cp, svn switch, etc, and worry about whether you branch on the mirror path or the local path and what effect that would have, etc. And put up with Subversion followers saying that was natural and easy. Whatever.
$ cp -a parrot parrot.my-branch $
Each of those copies is fully independent, as if you gave them to someone else. You can easily push and pull changes between them without tearing your hair out. But that was too slow and heavy. We want to create new branches at the drop of a hat (trust me on that for now, yer just do, OK?). Maybe you don't want to copy the actual repository, just make another checkout. We can use git-clone again;
$ git-clone -l parrot parrot.my-branch Initialized empty Git repository in /home/samv/.svk/parrot.clone/.git/ 0 blocks Checking files out... 100% (2815/2815) done $
The -l option to git-clone told git to hardlink the objects together, so not only are these two sharing the same repository but they can still be moved around independently. Cool.
But all that's a lot of work and most of the time I don't care to create lots of different directories for all my branches. I can just make a new branch and switch to it immediately with git-checkout:
$ git-checkout -b localbranch remotes/trunk $
But wait, you say, don't I have to enter a commit message for this new branch?
Well, a branch in git is just a pointer to a commit. If you look at "gitk" now, you'll see a new green label on the same commit as "remotes/trunk" called "localbranch". They're like little "post-it" notes - with a new enough gitk you can pepper your history with them wherever you like with a click and then typing the name in. They generally don't form a part of the permanent history - it's the actual commits, the changes to the code, that are the history.
Once you have some edits you want to commit, you can use git-commit to commit them. Nothing (not even file changes) gets committed by default; you'll probably find yourself using git-commit -a to get similar semantics to svn commit.
This is because git has a powerful concept called the staging area (old name: "index", hence commands like git-update-index), which is where you can prepare your changes before you actually save the commit.
$ vi CREDITS $ git-commit -a committed tree 6b513546099f01826c5cc7bc25042d00bc2560b0 $
Interactive commit is not really there, unless you use Cogito's cg-commit -p with this patch (which fixes the way that editing a patch for a change before it gets committed works). Normally people just use the staging area functionality, though. This is certainly one area where SVK's UI is better than git-core. But UI sophistication is usually only a temporary problem, especially for a project with as much energy pouring into it as git.
Update: oh, dear, I'm just so out of touch.
Did you mess up a change? Commit something poorly? Well, no worries, there are lots of ways to fix it.
Again, we're diverging from things that SVK supports well, but I think they're important to get a taste for how things are different. According to one source, lack of support in SVK for this is a "philosophical" stance. I really don't understand this - I make mistakes all the time and it's better that I correct the ones I catch early so other people don't waste their time on them.
If it's the top commit, you can just add --amend to your regular git-commit command to, well, amend the last commit.
You can also uncommit. It's such a crude thing to do that there isn't a command for it (if you add the cogito wrappers, you can use cg-admin-uncommit).
$ git-update-ref refs/heads/localbranch HEAD~1 $
HEAD~1 is a special syntax that means "one commit before the reference called HEAD". I could have also put a complete revision number, a partial (non-ambiguous) revision number, or something like remotes/trunk. See git-rev-parse(1) for the full list of ways in which you can specify revisions.
And just like that, your most recent commit was unlinked. If it really was garbage, that was what you wanted. Actually, it isn't completely gone;
$ git-fsck dangling commit 2ef718cf5434eeb8fdec74e69968f64fadd28761 $
If you wanted, you could see it with, eg, gitk 2ef718. I sometimes write commands like `gitk --all `git-fsck | awk '/dangling commit/ {print $3}'`' to see all the commits in the repository, not just the ones with "post-it notes" (aka references) stuck to them.
But that aside, uncommitting really is a primitive mode of operation, and you'd probably end up getting confused by the fact that git-update-ref didn't change the staging area. This is because git-update-ref is a plumbing command; it does one thing, and does it quickly and well. Commands like git-commit are considered porcelain - that is, designed for user interface. So, the technical name for the above dangling commit is spillage. This analogy doesn't seem to extend far enough to make git-prune (which would delete that commit) called something like git-flush or git-pull-chain, however.
Git's more of a toolkit for writing VCS than a VCS in its own right, and if you're the sort of person who doesn't like too many commands, then try Cogito - its simplified feature set is much easier for beginners. However, I'm not one of those people - git is like that little engine that one day you realised you could take apart completely yourself and understand what each part does (at least in principle). But I still prefer Cogito commands much of the time.
So, anyway, there are other tools for revising commits, and to be the king of patch revisioning is Stacked Git.
Say I discover a change that I actually wanted to apply three commits ago. Assuming that I haven't sent the patches out yet, then I can just go ahead and change them; no-one need know. In fact I can anyway, it's just that the longer ago you change things the more antisocial the behaviour becomes. In this scenario, we'll assume that what I'm currently working on isn't finished, either - and I don't want to have to finish it first.
$ stg init branch 'localbranch' initialised $ stg new -m "WIP." new-commit ... $ stg uncommit -n 3 ... $
Now, stg uncommit didn't do the same thing as cg-admin-uncommit; specifically, it didn't unlink any patches. They've just moved onto the patch stack, which I can jump around with using stg commands. First I'll extract the current patch with stg diff, edit it, then apply it a few revisions up.
$ stg diff -r /bottom > this_commit.patch $ vi this_commit.patch $ stg pop -n 2 now at patch 'do_something_interesting' $ patch -p1 < commit.patch patching file foobar.c $ stg refresh $ stg push -n 2 now at patch 'do_something_else_interesting' $ stg commit $ stg push now at patch 'new-commit' $ vi foo.c $ stg refresh -e $ stg commit $ stg clean No patches applied $
But this isn't a tutorial on stacked git. See the Stacked Git homepage for that.
"Another" way to revise commits is to make a branch from the point a few commits ago, then make a new series of commits that is revised in the way that you want. This is the same scenario as before.
$ git-commit -a -m "WIP." committed tree 5ef9339c5b5bc6572b69ff61cdb1dd4af4603f0b $ git-checkout -b tempbranch HEAD~4 $ git-cherry-pick --no-commit -r localbranch~3 ... $ vi foobar.c $ git-commit -a $ git-cherry-pick -r localbranch~2 ... $ git-cherry-pick -r localbranch~1 ... $ git-cherry-pick --no-commit -r localbranch ... $
This technique is called rebasing commits.
There are many, many ways to skin this cat. To tell the truth a lot of them don't play well together, for example you'd better remember to use "stg clean" before committing with something else. It's the old Cathedral vs. Bazaar thing. Using Git opens the door to a bazaar of VCS tools rather than sacrificing your projects at the altar of one. That said, these situations are usually easy enough to recover from in practice, especially by asking for help in #git on freenode.
If you pulled from my source, you can update the latest Subversion revisions I've put there using the native git command:
$ git-fetch ... $
This command completes very quickly even when pulling thousands of new revisions, modulo bugs for obscure corner cases like repositories with a huge number of non-overlapping revisions.
On the other hand, if you pulled from the Subversion Server - the slowest option above - or you are continuing to use SVK to do the real fetching (and have just run svk sync), you can just use:
$ git-svn fetch ... $
If you converted the repository from your SVK depot, and you don't want to continue using SVK, then the safest thing to do is first clean out the git-svn metadata; but look out for git-svn updates that do this in a smarter way.
$ rm -r .git/svn $ vi .git/config $
If you copied the repository somewhere else (eg, from me) via git-clone, then you won't have any SVN metadata - just commits. In that case, you need to rebuild your SVN metadata, using the same command as in the earlier section, but with the upstream URL:
$ git-svn init -t tags -b branches -T trunk \
https://svn.perl.org/parrot
...
$
You should see a stream of messages saying "r1234 = e79d0a84830becb10f6f6d24a9e0b7e3663c2921" (etc) as git-svn scans through the commits and makes its index. After you've rebuilt the index, the above git-svn fetch command should do the trick.
The recommended way to do this for people familiar with Subversion is to use git-svn rebase. You actually don't need to use git-svn fetch separately; it will automatically fetch new revisions first.
$ git-svn rebase ... $
This command is doing something similar to the above commands that used git-cherry-pick; it's copying the changes from one point on the revision tree to another, just like svk sm -Il.
The command to use is git-svn dcommit. The d stands for delta (there used to be a git-svn commit command that has since been renamed to git-svn set-tree because its behaviour was considered a little surprising for first-time users).
git-svn won't let the server merge revisions on the fly; if there were updates since you fetched / rebased, you'll have to do that again. People are not used to this, thinking somehow that if somebody commits something to file A, then somebody else commits something to file B, both changes should survive despite none of the people committing having a local copy with both changes.
Again there are lots of ways to do this. Let's say we've made some changes and want to make patch files for all of the ones since trunk:
$ git-format-patch remotes/trunk ... $
A command like git-log remotes/trunk..HEAD would show you the commits that this involves. You can then take those patch files and attach them to e-mails or whatever.
If the project uses the kernel patch submission policy, which strangely enough is very similar to best practices for sending patches to usenet etc since 'patch' was invented, then you probably don't want to use --attach.
If the upstream applies your patch without changes, then if you later merge, the changes shouldn't need to re-merged. git will notice that there has been a revision since the "merge base" that an identical change was applied and realise it has already been done.
We've shown a lot of stuff so far that shows that git-svn can do everything that we expected of SVK in order to have a much better Subversion client. What else did we win?
I've already talked a bit about the fact that git is a toolkit for writing VCS systems. As a result, one huge benefit is a flexibility and wide range of tools to choose from. Writing a tool to do something that you want is often quite a simple matter of plugging together a few core commands. The git repository model is also simple enough that there are even alternate git implementations you can draw upon.
I've also talked about patch revising using stacked git, touched on rebasing, and I'm sure you can read between the lines that dropping commits is also possible.
Then there was the repository efficiency, which affects everything - the virtual memory footprint while mining information from the repository, how much data needs to be transferred during "push" and "pull" operations, and so on.
But really, what does git win you? For a start...
You can easily publish your changes for others who are switched on to git to pull. At a stretch, you can just throw the .git directory on an HTTP server somewhere and publish the path. You don't need any silly Web-DAV extensions built into the web server just to share revisions.
There are also sites like repo.or.cz which will let anyone start a new project (or publish their fork of an existing project).
There's also the git-daemon for more efficient serving of repositories (at least, in terms of network use), and gitweb.cgi to provide a visualisation of a git repository.
This means you can...
With Subversion, everyone has to commit their changes back to the central wiki, I mean repository, to share them. SVK claims to be distributed, but this is, at best, demonstrating misunderstanding of what being ditributed means. By almost all definitions SVK merely offers disconnected operation. If I meet you in the middle of a cruise and we both have a copy of a subversion repository, I can't easily share my local branch with you if we're both on SVK. Doing this has come up on the SVK list before to a resounding "dunno, never tried, might work..."
With Git (actually this is completely true for other distributed systems), it's trivial to push and pull changes between each other. If what you're pulling has common history then git will just pull the differences.
So I'd just copy my repository to a USB key, stick it into the target machine, then run:
$ git-pull /media/usbdisk/project.git ... $
Sure, a USB stick isn't as gimmicky as a peer to peer wireless protocol featuring autodiscovery. But frankly I'll put up with that for sane branching support in the first place.
If the person publishes their repository as described above, using the git-daemon(1), http or anything else that you can get your kernel to map to its VFS, then you can set it up as a "remote" and pull from it;
$ cat > .git/remotes/friend >>EOF URL: file:///net/friend/git/project Pull: refs/heads/*:refs/remotes/friend/* EOF $ git-fetch friend
Here we're configuring all of the heads (aka branches) of the repository which appears at /net/friend/git/project to appear as remotes/friend/XXX in our repository.
$ git-merge remotes/trunk ... $
If git's history-sensitive merging doesn't automatically resolve things like patches applied in a different order, you end up with conflicts. The local file gets conflict markers - which might sound apalling, but the "ancestor", "left" and "right" versions of the file are nearby in the staging area. I like to use ediff-merge-files-with-ancestor to merge, so my merge script handles starting this for me to make merging easy. And I don't have to worry about breaking out of a merge aborting the whole thing and throwing away work.
No doubt some will say that SVK's UI is better because it lets you make per-file decisions as you make the merge. I see that as an easily possible addition to the git-commit interface. It's just I've been quite happy to resolve using the facilities of the staging area.
When you look at the commits you make using git-merge in gitk, you'll see something interesting - the new commit has two lines coming back from it. It has two parents. It's something equivalent to SVK's merge tickets, except that the information doesn't become worthless when pushed back to a server.
This is an interesting one. Some repositories, for instance the Linux kernel, run a policy such as no commit may break the build.
Because you can easily separate your repositories into stable branches, temporary branches, etc, then you can easily set up programs that only let commits through if they meet criteria of your choosing.
You might use a continual integration server to check that no commit happens that breaks your build. You might say that no merge can happen unless the branch added tests, and that tests pass. You might say that commits either have to add tests or make tests pass.
Your "trunk" becomes merely a point where branches considered stable are merged into. Each of your feature branches can merge from the trunk easily, which means that an immediate merge back in the other direction will involve no actual changes (and, in fact, no extra commit will be made in such a case - the head pointer will just be moved).
Bazaar comes with some great utilities like the Patch Queue Manager which helps show you your feature branches. With PQM, you just create a branch with a description of what you're trying to do, make it work against the version that you branched off, and then you're done. The branch can be updated to reflect changes in trunk, and eventually merged and closed.
Your SVN server going down doesn't kill your team's group development if people use systems like repo to mirror and track each other's repositories. They just stop pushing to the published branch and push to each other for a bit.
Of course if I didn't mention these then I'd have people ranting about how I was biased and partisan etc. But there are many shortfallings in git.
Not least is that it doesn't support two popular styles of developing
This is where instead of merging in patches completely, you merge bits of them in on a file-by-file basis, and expect the VCS to tell you what you did.
This is where you send new features into the ghetto so that they can 'battle it out'. The last features standing get re-integrated into another branch known as the trailer park to try to find a new life for themselves.
Note that ghetto is frequently called trunk, and the trailer park something like releng.
We have the tools we need to break away from centralisation! Now, we just need to convert the 10,000 projects...
Earlier in this article I referred to history rewriting in passing. I include this as a pointer for the keen, but bear in mind that this falls into the class of "history munging", and for various reasons is best done in the privacy of an unpublished project.
Let's say that we have a branch (the current one) that contains all the patches that we want to move to a rebased history.
We manually find a common commit (possibly using gitk). Let's say it was commit 7cbf53525bc6387495edd574ecdb248e1e4f872a, which became aa3e7febb0477e15257c89126d037f6f81a7974c. You'd re-write that using the cogito command:
$ cd-admin-rewritehist -k 7cbf53 \
--parent-filter "sed -e 's/7cbf53525bc6387495edd574ecdb248e1e4f872a/aa3e7febb0477e15257c89126d037f6f81a7974c/'" \
new-branch
That's a one-line history graft. You now need to go through all of your refs that point to the old commit IDs and point them at the new ones.
Be careful with this kind of history munging, you might just end up with somebody wondering why their "git-pull" is taking so long to negotiate which commits it has and hasn't got.