How to anger your customers

A few days ago Ian Landsman posted on the subject of distinguishing between features you don’t have because they are inappropriate for your product and features you don’t have out of sheer arrogance. We recently ran into a case of the latter that has cost us many man-hours to deal with. That feature is label support in Perforce.

But Perforce has labels, you might say. Well they have labels, but they don’t have labels that you can use like you would in any other version control tool. If you try, you will end up with a dog-slow Perforce repository and a bunch of unhappy employees. Perforce supports labels, but they simply don’t scale to real-project sizes. We found label support in Perforce and, assuming that they had support for them, set up the daily build script to operate something like this:

  1. Get the latest version of all the files out of perforce
  2. Build the game
  3. Label the version of all the files that were synched at step 1 with the version number of the build

This is the same build process I’ve used in RCS, Starteam, ClearCase, and Visual Source Safe. It’s just what you do. You can use that label to sync a very specific version of all the files out of the repository, and we often do. Labels are central enough to the Perforce experience that there’s an entire menu dedicated to them on the top of their UI. The only clue we had that labels might not be the best choice in Perforce was this note from the label documentation:

Before you assume that a label is required, consider whether simply referring to a changelist number might fulfill your requirements.

Well fast forward four and a half years and run a build every work day. Over that time our Perforce repository accumulated over 1100 labels. That’s certainly a lot of labels, but since almost all of them were sitting inert in the database, it didn’t seem like a problem. But over the past 4-6 months our Perforce performance has really started to bog down. As of last week, simple operations were taking 5 minutes or more. And by “simple” I mean refreshing the UI.

We tried upgrading our Perforce server first. It was 4 years old, so that seemed like a reasonable explanation, but it actually made the problem worse. When we finally called Perforce support to find out what was going on, they were shocked that we had the gall to use labels. After all they, only included them in the product to ease the transition from other version control systems, not because anyone was supposed to use them. At the end of the support call, after rolling back to an old server version and occupying one programmer and one IT guy for an entire day, they sent an email that included this gem:

You will see better performance even before you delete the labels — using changes instead of labels is far less intensive in terms of resources. Also, they may grumble about it now, but I suspect your developers will be much happier when they realize that they don’t have to create and maintain labels any more — the change number system is automatic.

Excuse me? I don’t “have to” create and maintain labels now. I type “build” and the labels just happen. The level of “our way is better, and we know what’s best for you” in this email is astounding. It mirrors earlier comments made over the phone by the same support guy.

Now, I don’t really care about the labels vs. changelist numbers debate. They want us to use changelist numbers, and if we had known labels didn’t really work from the start, we would have written our build system around changelist numbers instead. What offends me is that a feature they display so prominently is something we aren’t supposed to use, but they couldn’t be bothered to tell us this anywhere in the documentation. If labels are such a problem for them, they should put something like “Labels have serious performance implications for large repositories. Use changelist numbers instead.” somewhere in their documentation.

And what’s with the attitude? We paid nearly $40,000 for this product (for our 50 seats.) We pay them $8,000 per year to keep our license up to date so we can add new licenses when we need to. That’s an awful lot of money to pay for them to tell us in a snooty voice we’re wrong for using a documented feature in their product.
Yesterday I wrote a quick one-off script to delete all the build labels that were more than a month old and which didn’t point to a version of the game we ever distributed or showed to anyone. Now that those are gone, our Perforce performance seems to be back to normal. We may eventually switch to changelist numbers, but it will be on our schedule, not in some sort of emergency fix mode. We’re also looking around at other revision control systems that won’t look down on us as much as Perforce apparently does. Anybody have anything to say one way or the other about Accurev?


12 Responses to “How to anger your customers”

  1. Stephen Taylor commented on :

    By golly that is a lot of money. But at least they give you good support…

    I really worry about how some people earn their pay cheques. I’d understand if he was a bitter I.T. guy, sick of restarting printers…but really, a support guy being so snooty is truly shocking.

  2. Christian wrote on :

    I can tell you two things for certain that AccuRev does not have. First, they do not have support people or engineers who will give you the snooty voice. They are very nice folks and always quite helpful! Second, they do not have labels in any form. As an old Visual SourceSafe shop, we are using a combination of snapshots (for recording Alpha, Beta RTM, and SPs) and transaction numbers (for replacing per-build labels). We’ve been very happy with AccuRev at DeLorme. I wish you the best with Perforce.

    Cheerio from apparently sunny England,

  3. Michael Dubakov thought on :

    Why don’t you use Subversion? It is great SC system, pretty fast and stable. We are using it for 2 years without such problems and it supports labels just good :)

  4. Petros Amiridis replied on :

    I don’t want to sound a wise guy, but what are the benefits of using such an expensive version control system instead for example Subversion. Is it worth the money? This is an honest question, since I have been using Subversion and haven’t tried another version control system.

  5. Joe replied on :

    The two big reasons we’re not using subversion are:

    1. Half of our users are artists. Perforce is already techy and scary to them. Subversion is even more techy and scary.
    2. I hadn’t heard of subversion when we started the project in 2002.

    I assume there are some GUI front ends out there that can help with #1. The “edit whatever you want and we’ll figure out what you changed at commit time” aspect of subversion certainly helps all by itself. (Though I fear the number of accidental changes they would submit to the depot.)

    And of course only a time machine can change #2. :)

  6. Bill said on :

    We have had no problems at all with Perforce technical support, in fact they have always been the most responsive and helpful of our tool vendors.

    We investigated Labels and Changelists when setting up a continuous build system and quickly determined that changelists were the way to go.

    Perforce Labels are used to tag a set of files with the version of each file specified individually, not just the latest tip of your development branch.

  7. Joe commented on :

    Labels are nice because they can be used to tag any set of files with the version of each file specified individually OR to tag the set of files in the repository at a specific point in time. There have been times when labeling a build as “this previous build plus this one changelist” has been handy. Of course most of the time a changelist number would be just fine and we will probably switch over eventually. If we had known labels were such a problem we would have never used them in the first place.

    My complaints are mostly about the crappy attitude we got from support, not the labels vs. changelist numbers debate. It’s usually better to work with your tools instead of against them, and using labels in Perforce is certainly going against the grain.

  8. Cam wrote on :

    Subversion is brilliant, Previously I used CVS but now use Subversion which owns it.

    Eclipse is my favorite way to use Subversion.

    There is also a free Explorer Subversion plugin called Tortoise.

    But yeah why pay money for something that ain’t cutting the mustard especially when there are better free alternatives?

  9. Joe replied on :

    Whether the free alternatives are “better” or not is certainly debatable.

    FWIW, we’ve now moved perforce onto a new server and all the performance problems are gone.

  10. Lars thought on :

    Interesting. I work in the financial industry and we had one group make the migration from ClearCase to Perforce partially because our ClearCase servers bogged down due to “too many” branches and labels.

  11. Joe said on :

    The environment I was using ClearCase in was at HP working on an embedded system. We had about 300k lines of code total and only a couple branches. We also ran weekly builds rather than the at-least-once-a-day builds we run at FLS. I guess our number of labels wasn’t large enough to cause trouble.

1 Trackback

  1. Trackback from Ian Landsman's Weblog v2.0 on :

    Feature Arrogance

    Joe has an interesting post about feature arrogance, relating to my post the other day on features. His story is also a good example of poor customer service, especially for a product with a $40,000 price tag. Hey, maybe it's time to raise my pric…