Introduction

• Good morning, I'm Keith Bostic.

• To give you a little background of where I'm from, I spent some time at the Computer Systems Research Group at the University of California, Berkeley, working on the 4BSD software distribution, and I've spent some time at Berkeley Software Design. That means that I've been around Open Source for longer than it's been called "Open Source". I've worked on a number of large "Open Source" projects, including 4BSD, Berkeley vi and Berkeley DB.

  I currently work for a company named Sleepycat Software. We are a completely distributed company and we make our money by licensing and providing support for the Berkeley DB database system.

• I'd like to thank you all for inviting me. I appreciate it a lot, and I take it as a compliment that you think that there's something useful that I can say to you.

• When your program committee invited me to speak, I was pretty hesitant. But, a few people did some arm twisting, and I think the matter was finally settled when the conference brochures were distributed with my name (albeit misspelled) listed in them. Frankly, I usually avoid doing these kinds of talks, for lots of reasons. I didn't want to do the "What was the CSRG like" talk, or the "What is Open Source" talk, or "The embedded database market" talk. Anyway, after some thought, I decided there were some things that I would really like to say to this group, so I do want to thank you for giving me that opportunity.

• Today, I want to talk about the things that I've learned, or, at least that I think I've learned, in the past few years about Open Source, distributed development and businesses, and about running an Open Source engineering team. I believe this information is important to you folks, because whether or not you're trying to make money from your work on *BSD, you have most of the problems and concerns of a business -- you have chosen to measure your success in somewhat different ways.

• To some extent, I'm outside of your community (I'd hate to tell you how long its been since I compiled a kernel), and, so, if I say something that's wrong, I would encourage you to ask "Why does this person have this perception of our community", not, "What an idiot." (Of course, if the answer to the question "why does this person have this perception...", is, "well, he's an idiot." that's fine, you should feel free to stop me at any time.

• To no-one's surprise, I'm going to talk about some issues where I think the BSD community is missing the target. I don't mean to concentrate on the negative, and up front I'd like to say that, overall, I think that the BSD community is executing well. I'm optimistic about our future or I wouldn't be here -- I think that this community is going to be an exciting one of which to be a part in the coming years.

What this talk is about

• If there's an overall emphasis for my talk, it's "Open Source Engineering (or Technical) Management". I want to talk about how engineering gets done in the BSD community as a whole, within individual development groups, and little about your focus as an individual contributor.

  I'm going to use the word "focus" as much as "management". First, while it's not immediately obvious in many companies, the purpose of management is to help you get your job done. Better. Faster. Not get in your way. That's the kind of management I want to talk about. The kind of management that helps you determine what are the important problems to solve, and helps you focus on solving them. Second, well, people tend not to get so upset when I say "focus" instead of "management".

• The reason that I want to concentrate on this topic is because I think engineering management is the single item most likely to be missing from an Open Source project.

• There are four parts to my comments:
  1. I want to talk about why engineering management is vitally important for you.
  2. I want to talk about how to increase your group's focus, and the group's ability to execute.
  3. I want to talk about what I think we need to focus on in our development groups.
  4. And, I want to talk about what I think we need to focus on in a few specific technical areas.

Virtual Teams

• The first reason for increased engineering management, and, a strong argument for engineering management being *more* important to Open Source projects than other projects, is, in the buzzword of our times, "virtual teams".

• I'm sure you've heard the old joke: "Managing programmers is like herding cats." The problem with virtual teams is they are the worst example of the "herding cats" problem of which I'm aware. Distributed engineering only works if everyone is highly motivated, self-directed and disciplined. Since no group is composed entirely of people that are highly motivated, self-directed and disciplined, or, at least not all of the time, virtual teams are pretty much a disaster waiting to happen.

• Now, I realize that some of you have already assumed your sleeping-in-class faces and tuned me out with the simple response, "So what? That's *their* problem, I'm not a manager, I'm hacking on this stuff for fun." I'd like to correct you on that point before we go any further. You *are* a manager. You manage your time, you manage your teammates, and you manage your group leaders.

  You joined this group because you cared about its goals. In fact, since you aren't getting paid, I know that you care more about the groups' goals than most people care about their "job". And, frankly, as a group member, my point-of-view is that I don't want my teammates getting off track, because it lessens my chances of reaching my goals, and maximizes my chance of having wasted my time. You need to understand the engineering management issues in your group if for no other reason than to defend your own self-interest in getting your goals accomplished.

• I'd like to talk about the problems of virtual teams, at some length:

• Team members cross geographic boundaries -- they don't share a culture. And, hacker ethos isn't what I mean by culture. What I mean is whether or not Jane Doe considers email to the group saying "Jane Doe checked in an old version of the source code on top of my buffer changes and screwed the pooch" to be cause for a contrite reply, or a flat declaration that she is an outcast no longer welcome to contribute in any way.

• Team members cross economic and power boundaries -- some are high school students, some are programmers, some are CTOs that like to code at night. This is especially tricky for two reasons: first, it's difficult for people to adopt different attitudes when filling different roles, and second, cultures vary widely in power-distance, that is, how willing people are to correct other individuals at different power levels in the organization.

  As an example, consider a dot-com CTO working in a virtual group with a Japanese high-school student. The CTO is going to have a difficult time deferring to the student on technical issues. The student, coming from a culture with a strong power boundary, is going to have a difficult time telling the CTO that they're wrong. And, the whole group will feel the tension.

• Team members don't meet and they don't telephone each other, because they often don't work the same hours, and may not even speak the same language. (Many, many people that can write a foreign language cannot speak it.)

  Email isn't anything like a reasonable communications protocol. The success of emoticons is an absolute indictment of the ability of email to distribute information. Speaking as a man who has broken off relationships because my SO dotted her i's with little hearts, I find the fact that half the world puts smiley's in their email to be one of the principle signs that the Apocalypse is upon us.

• Team members don't work regular hours, and rarely work full-time. It is difficult, if not flatly impossible to design a new virtual memory system for a modern operating system in two hours a night and 4 hours on a weekend morning. The "we can do anything because we have a million monkeys hammering on keyboards" statement needs a slight bit of refinement, in my belief. It should be "we can do anything that doesn't require more than a reasonable amount of deep thought."

• It's almost impossible to mentor in virtual teams, and junior people are difficult to integrate. If you look at distributed companies, they tend to hire very senior engineers that people in the group have already worked with, simply because it's difficult to transmit a corporate culture and mentor a junior hire.

• In virtual teams, "Team managers" (or, perhaps, "group leaders" is a better term) have less, or at least different, leverage because they're not paying the bills. If you don't have a way to negatively motivate people, it's difficult to make them something they really don't want to do.

• In a virtual team, it's impossible for a "manager" to walk down the hallway and get a sense of where his team is in a few minutes. One of the best managers I ever had would drop by my office in the afternoon and our group would trek down to the office building candy store, and we would all buy Mountain Dew and a bag of M&M's, and then hack like crazy on a sugar high into the late night. That manager "hated meetings", or so he said. What we were too young and stupid to realize was that he was not only having a meeting every day where we reported our progress, but he was getting us to skip dinner so we'd work extra hours at the same time.

  To keep the cats going in the same direction, you have to know where they are, and you have to know when they've wandered into the weeds. In a virtual company, replicating that simple 10-minute walk-and-talk down the hall requires hours or days of exchanged email, telephone calls and so on.

  Even meeting physically once a week won't give you the kind of synergy that you get from a team working at the same location. And the technology for virtual meetings isn't widely available yet, and, even if it were, current indications are that it's not going to be a sufficient substitute for physical meetings anyway.

  The kind of "meetings" that work best are the entirely unplanned opportunities for discussion in the hallways and coffee shops, or where you get dragged into a room with a whiteboard to work through a new idea.

  At BSDI, we had a saying: "The best thing about a distributed company is that you never have meetings. And the worst thing about a distributed company is that you never have meetings." At the CSRG, if there was a policy decision to be made, everybody went into a room with a whiteboard, shut the door and in 20 minutes, well, there might have been unhappy people, but we had a direction in which to proceed. When working in a virtual team, I've seen 20 minute engineering decisions take weeks, and I'm not exaggerating even a little bit.

• Since members of a virtual team often work on their own most of the time, they tend to get much deeper into the weeds. The scenario goes something like this: someone has an idea, they send out some email, nobody replies for a couple of days, and by the time a reply is sent, the idea is pretty well entrenched, and there may already be code that implements it. Waiting a week to pull a bad idea out of the code is much less efficient that killing the idea during lunch one day.

• In virtual teams, you can expect email flame wars to last a lot longer and make far less sense. The scenario here goes something like this: someone gets annoyed, and sends some email. There's no reply, so he sends more email, each time getting more and more irate, while ignoring the fact they're angry with someone who won't be reading their email for another day, at least. Then, when the recipient finally *does* read their email, they reply to the first flame before reading all of the messages, which leads to several subsequent rounds of "I already replied to that, but since you clearly didn't understand my arguments, I'll say it all again."

• I'm not saying that you can't make virtual teams work. They can be made to work with enough brute force and a few team members that are politically savvy and who are willing to pay attention to keeping everything running smoothly. It's certainly not the easiest way to do things. If you can create a physical center for the group, I strongly recommend it.

• Absent a physical center, the best answer you can make to the problem of virtual teams is an increased focus, and therefore efficiency, in your group. The goal is to increase your team's percentage of correct decisions, when that decision is made without additional input from other group members.

Time

• The second reason for increased engineering management is that you need to focus to maximize your time. I want to briefly address this, because the less focus you have, the better the chance that you're wasting your time.

• Open Source teams have less time than the groups they're competing against because team members don't work full time, and their Open Source work isn't their first priority. Which makes the time you have more precious than it would otherwise be.

  1. First, you simply don't have that many hours to waste. In my experience, volunteer coders work 8 hours over the weekend, but sporadically during the week, and they tend to disappear periodically.

  2. Second, the latency kills you. It's easy for a team member to hold up another part of the team for weeks, simply because they don't answer email for a few days at a time.

  3. We also all know that the more vital someone is to a project, the more likely that they're busy with other things. The most important people to a project are usually the busiest.

  4. Finally, remember that we're talking about engineers. How do engineers deal with being overloaded? All together now: they work *harder*! That's short-term thinking, and all it leads to is worse bottle-necks down the road, because now your decision makers are not only overloaded, they're burned out.

• By increasing the focus of your group, you're going to decrease the likelihood that time your group has to invest is wasted, and increase the chances that you fulfill your goals.

Mission statement

• But, you say, "My group is focused. We know exactly who we are, what we're doing, where we're going, and who we'll see when we get there. You're not talking to us".

• I won't disagree with you, some Open Source groups are amazingly focused. But, even though I'm sure that every group represented in this room is focused like a laser beam, since we're already here and we've already paid for a few more hours of this convention hall, let's move away from problems that increased engineering management might help you address, and talk about a couple of ways you might be able to increase your group's focus.

• Do you have a mission statement? Yes, I know we all laugh at Dilbert when he and his colleagues read various corporate mission statements, but that doesn't change the fact that the single most important thing you can do for your group to improve your focus is to put together a couple of sentences (no more!) that spells out what you're doing. Using the word "empower" is a bad sign, as is using more than two commas, or words of more than two syllables.

• A good mission statement is something like: "Build a secure, standard UNIX system." There are three goals here: secure, standard, UNIX-like. Clear, concise, to the point. Now... go to an average Open Source web page on sourceforge, and try to find a mission statement. It doesn't necessarily have to be described as "mission statement", but find the information. Find a paragraph that says, "We make widgets that do X, Y, and Z." You'll find lots of pages that say "About the Foo project." "Contributors." "New features." "Press". You almost never find the mission statement. Now, I'm not saying it has to be on your web site, although I would encourage it. But it has to be well-known to your group.

• Once you have a mission statement in place, it means that everybody on the team, when confronted with a decision, can make the decision with some likelihood of being correct. "We have been approached by company XXX to incorporate new drivers for their sound cards", and that's going to take real resources from the project. Easy decision: it doesn't get us any closer to our mission, as defined by our mission statement. We either change our mission statement, or we don't do the work.

• The other thing that groups without a mission statement find, when they try and create one, is that everybody disagrees on what the mission is. That is, they were all doing different things, and they didn't know it. The internal tension in those groups is often uncomfortably high -- it's maddening when people are wasting your time, only it turns out they're not wasting *their* time because they had a different set of goals all along. You often see nasty, ugly breakups in those groups too, when the tension gets too high.

  A book I read some time ago had some wonderful case studies of companies trying to create mission statements. My favorite was the company where an N-million-dollar business unit was actually split into two geographically separated parts -- it turns out the group really had two completely different goals, and resources and people were getting frantically shuttled around trying to keep things in balance.

• Another thing about creating mission statements is that I recommend having them throughout the organization, not just at the top. The top one may be "Build a secure, standard UNIX system", but the SCSI device driver group is going to have a different one, compatible with the overall goals of the group but more specific to their part in that work.

• Finally, think about it this way: how many of you go to work every day without a clearly defined goal for the day? Very few of you, I suspect -- you're all highly motivated individuals or you wouldn't be here. Why does it seem the most natural thing in the world to have a personal mission statement for the day, but not have a similar direction for your group as a whole?

Business Plan

• Now that you have a mission statement, do you have a plan? A business plan, that is. If the mission statement describes what you're going to do, the business plan tells you how you're going to do it. Now, when you start looking around for sample business plans, you're going to see a lot of verbage about venture capital, attracting investment and so on. Most of that isn't likely to be applicable to Open Source projects, but the rest of the information will be.

• Simply put, a business plan tells you, over the next year, and sometimes over the next few years, specific actions you're going to take to accomplish your mission statement. It doesn't need exhaustive detail, the goal is so you, and the rest of your group, have a document that you've all agreed to, that you can use to guide decision making.

• Again, when you're working in virtual teams, there's an incredible lack of focus simply because of the organizational structure. You have to be aggressive about attacking that problem. Creating a business plan has three terrific results:

  1. Everybody agrees on the direction you're going to take over the next year.

  2. Everybody signs up. Everybody has said "Yes, this looks reasonable, I'll do my part."

  3. As people join the group later, there's an easy way to bring them up to speed. "Here's what we're doing, here's how we're doing it, what part would you like?"

• I want to explicitly note that a technical design document is not a business plan. In technically focused companies and most Open Source groups, it's seductive to think that if we build a good piece of software, they will come. And while that technical design and implementation is probably the most important thing the company does, it doesn't define the overall plan for the group or company.

• As a side note, I'd also like to take a minute to laugh at companies that don't make their business plan available to their employees. It's kind of refreshing, in a sick way, to see a company that admits it doesn't want the employees to know what they're doing.

Process

• So, now you have a mission statement and a business plan, or maybe you've always had them. You know, it's kind of embarrassing if you think about it, so far, all I've told you is that you should agree as a group on what you're doing and how you're going to do it. The problem may be that this is all *too* simple.

  We're all smart folks, right? And we all know what we're doing, right? There's a joke about the fact that every committee with N people has N - 1 agendas, because somebody has to take notes. Don't forget that, and don't assume that because all of the people in your group are bright, motivated people that they all agree on what the group is doing and where its going.

• So, the next thing I want to say about increasing focus is that you should increase your process.

  I want you to know that that I find it encouraging that nobody in the audience screamed and held up a religious object to ward off the forces of evil when I said "process". Usually, when I talk about increasing the level of process in an organization that's what happens, and somebody tries to shove a stake through my heart when I'm on my way back at dawn to sleep in my coffin.

• Let me emphasize that when I talk about increasing the process in the organization, I do not mean that you should increase the bureaucracy. It's easy to tell the difference between the two: process means that it's easier and faster to do your job right. Bureaucracy means that someone else has to become involved for you to do your job. Increasing the process in your organization should have some important results:

  1. There are fewer decisions to make. Why is that good? Well, remember what I said before: the decision makers in your group are the people most likely to be too busy, and giving them another decision to make almost certainly means that it will be some number of days before you get a reply to your email. It's always faster to not have to make a decision at all.

     And, fewer decisions to make should mean fewer ways to screw up. If there are fewer decisions to make, there are that many fewer decisions that can be made badly.

  2. You can use process to lower the barrier to doing unpleasant tasks. At one point in my career, I spell-checked the BSD distribution man pages. It was truly a once-in-a-lifetime pain, and it took me most of a week.

     But, as part of doing it, I created a list of "OK words", that is, words that were used in each man page that were OK, regardless of what spell said. Then, I created a couple of simple commands: one that updated the list of OK words, and one that did a spell check and compared the output against the OK list, kicking out any words that it finds that weren't in the list. From that time on, I could easily spell check the system over night, and, what's more important, I could get other people to do it to -- there no reason not to check if it's easy to do.

  3. Process helps smooth over the rough edges where parts of the project meet.

     We're all smart software developers and we like to think that we're good at elegance and precision. However, that's only true in our own areas. If you look at our projects, they're often sloppy globally. Features are released without documentation, significant changes are made in the code after initial release, code is only tested in the field, important fixes are backed out in later releases because the current developer doesn't understand why the change was made in the first place, and there's no test suite. Process can solve these problems.

     The trick is, everybody has to take their medicine or it won't work. The group needs to set up simple processes, and once they're in place, everybody has to follow them. This means that there's an absolute requirement that the processes be simple enough to follow and that the advantages of following the process be clearly evident. Everyone in the group has to understand that process is good, and agree upfront to put it into place.

     You also need to make the process part of the project infrastructure. People join and leave Open Source projects over time, and new members need to understand, abide by, and improve the processes that are in place. It's important for the leaders in the project to advocate process, to reward people for putting it in place, and to encourage everyone to use it.

  4. Finally, process can ensure that if you get hit by a bus, the group doesn't disintegrate, or, almost as bad, run in circles for 6 months trying to figure out who owns what, and who was responsible for fixing that bug anyway?

     One of the interesting things about the recent Verizon strike was that mid-level managers were on the streets, manning the phone trucks. Think about your organization: does everybody know roughly what everybody else is doing? Does everybody know where the important information can be found?

     It's possibly apocryphal, but the story I've heard is that at one time, everybody in the phone company spent time on the trucks, so that in times of emergency, everybody could be used out in the field. And, the story goes that Dennis Ritchie has apparently spent time climbing telephone poles. So, ask yourself this question: can everybody in your organization climb your particular telephone pole?

     I don't recommend that groups randomly swap jobs around, although some job swapping helps keep people engaged and learning, and happier with their assignments, not to mention more interchangeable. As a simple rule of thumb, I want to be able to replace anybody on my team, with someone else from the team that has at least a reasonably good idea where the bodies are buried. To do this you'll have to plan assignments and code review so that people can become at least passing familiar with parts of the code base other than their own.

• Process should give people more time, not take it away. The goal should be to automate tasks, so that people don't spend their time thinking about them. It needs to be blindingly obvious what to do in every single circumstance, and simple to do it. That's your goal.

• You should also introduce process quickly rather than waiting until there's no other choice. It's better to err on the side of introducing process too early rather than waiting too long. For example, why doesn't the UNIX kernel automatically create the right set of nodes in /dev for the available devices? I suspect that millions of man hours have been spent trying to figure out the magic mknod command to use on some random system to get some random tape drive to work.

  I personally have spent dozens of hours, not even counting that DAT drive the FreeBSD SCSI subsystem was treating in some oddly blocked way. In every organization I've worked with, I've complained loudly about this, and in every one my complaint has been answered that there are too many device drives to convert, and finding the answer for any individual drive isn't *that* hard. That argument is certainly correct, but not particularly satisfying -- imagine if Version 7 UNIX had had this process in place, think how much better UNIX system administration would have been for the past couple of decades.

  Put your process in early, and expand, or remove, it as necessary.

  The longer you wait, the harder it is to fix. Another way to look at this is that you are, by definition, putting process into your organization. It may be bad process, it may simply be "Bang your head against the wall until it works", but it is process. It seems better to me to put in good process rather than bad.

• How do you increase the process in your virtual team? OK, here are some specific suggestions for your group.

  1. Create an internal web site. If you don't have one already, you're missing something important. That site is where your business plan goes. That site is where your organization's hierarchy chart goes. That's where you find all of the email addresses and phone numbers for the group. The list is limitless: planning documents, release snapshots, source code style guide, company benefits, expense reports, FTP bug upload area, the list of conferences at which you're going to display, there's just no end to the possibilities.

  2. Maintain your software and other important information on a separate server (for example, sourceforge) that you know is backed up regularly. Because you are a virtual team, you have an absolutely splendid opportunity to screw up -- I can almost guarantee you that 50% of your organization isn't doing proper backups. And 85% of them aren't doing off-site backups. If somebody's house burns, are you out 6 months of work?

     The story goes that Bill Joy implemented multiple windows in an early version of vi, so, around 1980 vi had multiple window support. The code got lost when a disk drive crashed, and nobody had done backups. So, for lack of a single backup tape, the UNIX world lacked a significant feature in a significant tool for 15 years. ASPs are a wonderful way to solve this problem -- there are several that offer free or inexpensive software and marketing support to Open Source organizations..

  3. If you answer more than a couple questions on a topic, create a canned email reply, put it into the FAQ or the iweb site. Make your FAQs searchable by keyword.

  4. Add a bug report form, bug tracking software, make your bug reports searchable.

  5. Create project lists, and assign projects to people. It helps you focus on what needs to get done, and it helps you track your progress to a release.

  6. Some process looks an awful lot like marketing: create a vendor partner program, and make it available on your web site. Have recommended hardware lists, and example hardware configurations with references to hardware companies that will create it for you. We're part of a booming economy right now, which means that money is cheap and time is expensive. The easier you make it for people to use your work, the more of them will.

  7. Some process looks an awful lot like testing: take the time and trouble to put in an automated testing system. One fact we've learned in the past couple of decades of software engineering is that "If it hasn't been tested, then it doesn't work."

  8. Write the documentation, or, have everybody in the group contribute $100 and pay someone else to write it.

  9. Support general tools for the community: we don't have good run-time checking tools, we don't have good bug-tracking software. We need to start the projects that will get that work done, but I'm not holding my breath. The first is just hard, and for reasons that I don't understand, apparently the second is as well. Regardless, bite the bullet and find yourself a Purify license, and run Purify, or similar software fault checking tools on your software.

  It's hard to list many specific items, because each team has its own needs, but you know what I mean.

• Process also allows you to use people that are less engineering oriented to good effect. Have someone in your group do daily searches for references to your software in USENET postings. Better yet, have them start a newsgroup and moderate it, making sure that each question gets answered. This helps people get over the initial learning curve in using your software, and immeasurably helps its adoption. When I look at who is answering questions about various pieces of software on the net, it makes me want to cry. It's the architects, the authors. And it's because they care -- but it also means that they've failed to create the right team effort where people with different skills and skill levels are all working as efficiently as possible.

  For those of you who are about to comment that I'm the guy always answering questions about Berkeley DB on USENET, well, parts of this talk should have a note saying "Do as I say, not as I do."

  And, remember, experience is that thing that enables you to recognize a mistake the second after you've made it again.

• Finally, make a conscious effort to avoid short-term thinking. It's always easier to do the work yourself and not delegate, in the same way it's always easier to respond to a single email message than to write up the FAQ. But, it's fundamentally the wrong choice to make.

Decision making

• Another goal you should have is to push decision making (in particular, engineering decision making) as far down the chain as possible.

• One of the biggest faults in the Linux community is the fact that decision making tends to happen at a high level. The code base for a UNIX-like system is so large, now, that it simply cannot be handled by a small group of people. No matter how good and how well-intentioned people are, they are not going to be able to handle all of the details required by a full system distribution, and they eventually become bottle-necks.

• An advantage of Open Source is that the latencies introduced by geographic distribution of the team *force* Open Source groups to push decision making down the chain. This is good, because given that work expands to fill available time and more, you're going to have to push decision making out sooner or later. If you do it sooner, you usually get to do it in a controlled way. If you do it later, it's because someone has decided to actually spend time with their children, a release is 8 months late, and people are screaming for action. This scenario usually results in bad feelings all around, too, as one group of people rips the steering wheel out of the grasping fingers of another group of people.

• Another good thing about moving decision making down the chain is that you can avoid what C. Northcote Parkinson refers to as "bike shed discussions". Parkinson shows how you can go in to the board of directors and get approval for building a billion dollar nuclear power plant, but if you want to build a bike shed you will be tangled up in endless discussions.

  Parkinson explains that this is because a power plant is so vast, so expensive and so complicated that people cannot grasp it, and rather than try, they fall back on the assumption that somebody else checked all the details before it got this far.

  A bike shed on the other hand ... anyone can build one in a weekend. So no matter how well prepared, no matter how reasonable you are with your proposal, somebody will seize the chance to show that he is doing his job, that he is paying attention, that he is *here*.

  Richard Feynmann gives a couple of interesting, and much to the point, examples relating to Los Alamos in his books, as well.

  As decision making gets done farther down the chain, then there are fewer decisions that come to the attention of many people, and the group spends less time thrashing.

  But, you say, what if my junior person, who I've given this decision too, messes up? It just doesn't matter -- I'll talk about that more in a minute, but the bottom line is that we'll fix it in the mix, we'll fix it next release. It just doesn't matter. And, your junior person will have learned a lot, and, over time, they will make better and better decisions. If they don't, get yourself a new junior person. That said, people that volunteer to do work rarely need replacing. They usually are motivated enough that they want to get the job done right.

  The problem is that we're engineers, and we hate to give up control. We'd rather give up a kidney than let someone else make decisions about our code. And, in your groups, this isn't a job. These people are doing it *because* they like it, and that means they're going to care. Part of the reason that you've been stressing your mission statement (remember that?) is because it gives people something to focus on that allows them to agree that the details are less important.

  People will find a goal to pursue, it's human nature. If you go home feeling like you've made a difference, you feel good about yourself and the project. If team members don't have an overall project goal, they will instead focus on the smaller engineering goals, and that's precisely what you want to avoid.

  If you still can't give up control, keep reminding yourself that it doesn't matter how you solve most problems. I submit to you that in this room we have about ten people that have modified the source code for the UNIX utility "ls". And that's just wrong. Nobody out there could care less about the ls utility -- all of us could replace ours with the one from FreeBSD, BSDI, OpenBSD, NetBSD, GNU or Linux -- it just doesn't matter. Part of focus is understanding what the real problem is, and making sure that you're putting your time in where it matters.

  As specific suggestion for this audience, I think that we should all adopt a single port/packaging mechanism, pick a junior-level maintainer for 90% of the utilities in the system, and start treating the standard utilities in the same way we do the "port" utilities. Because the bottom line is that it just doesn't matter if we get the ls utility "right" or not. People don't care, and it just gets in the way of focusing on what we're really trying to accomplish.

• One last note of caution -- you can push the decision making down the chain, but at the same time, management by abdication doesn't work. You need to make sure that nobody gets left alone for too long, and that there's general chatter between team members. Email is probably fine, but I'd recommend periodic conference calls. Specifically, organize a once-a-month conference call that everybody can join in, it's not that expensive. More frequently, send out status email, I'd recommend once a week. Organize a beer party for the group every 6 months, for $5 at the door, and anybody coming from over 100 miles is free.

  And let me be clear: it doesn't matter what you say in the status reports or at the party -- talk about your latest big deal that everybody knows about, some milestones that you hit, a terrific bug fix by a team member, whatever. The goal in both the status report and the phone conversation is to establish the sense of community and culture, and shared goals that are essential to a good software team. Everybody has got to feel like they're pulling toward the same goal and that a success for the team is a success for everyone on the team. I'm not going to go into a lot of detail here, there are a million books on management and team motivation out there. You know what you need to do here, and I'll bet you anything you like that most of your teams aren't doing it.

  And finally, here's a quick organizational test: if you aren't doing status reports because "it's too much work", or, "it would take too long to pull together the information", that's a good indication that you're overloaded, and you need to start moving more decision making and tasks to other people.

Modularity and simplicity

• Next I'd like to talk about overall design goals for your software. First, modularity and simplicity.

• As I may have already mentioned, I think it's a demonstrable fact that, if a line of software hasn't been tested, it almost certainly doesn't work. Software reliability depends on testing, and therefore, on modularity, because the software code base in most projects is too big to depend on anything other than module testing.

  One of the folks at Sleepycat, Mike Olson, has a great analogy for this. You've got people machining bolts, casting the engine block, and wiring up the starter motor, but at the end of the day, someone needs to be sure that the car runs. Good engineering practice is to do unit and system testing. Everyone hates writing tests, and so many pieces of software don't get adequately tested until the car is on the freeway doing eighty miles an hour with a family of four inside. Failure is cheap at the factory and expensive on the road. It's important to make sure that you fail at the factory.

• There's also the issue of the initial programmer learning curve. As Open Source projects, depending on virtual teams, we have our own special problems in this area: we often don't get people for long periods of time. They'll join the group, do some work, disappear for 3 months, do some more work, and so on. In order for us to effectively use their time, we need two things: fast startup, and modular tasks. We can't afford the time investment for someone to become acquainted with a large code base and complex, undocumented interfaces. For this reason, we need to make sure that our projects are written both simplistically and in a modular way.

• The value of simplicity is what I think is the hardest sell to most engineers. As a group, we sure like our tricky little algorithms. I submit to you that we don't need those algorithms 90% of the time, and when we do, we should insist that they be encapsulated in library-style modules that can be separately maintained and debugged.

• We need to insist on compartmentalization. On documented internal interfaces. When you're working in a virtual team, where engineers drift into and out of the project over months, you need to invest the time to get the process in place. You need to invest the time to rigorously specify the interfaces, to spell out in complete design documents where the project is going. Otherwise, you can't divide the work or the responsibility cleanly, you can't maximize your development time, and when someone leaves the group, you thrash.

  I want to use this as an example for my earlier comments on process. Various groups, mine among them, have been threatening to write man pages for the BSD kernel modules for the past 15 years. What if it was standard practice in 1980, that adding a new kernel module to the system caused a man page describing it to be written, as we do for new utilities?

  Would the world be a better place right now or not? Would your programmers be better off or not? The reason it's happening only sporadically now is because doing it well would take 6 months of a senior engineer's time, and it's not going to be much fun. Putting process into place early can involve putting in process that you didn't really need, but getting process into place early will pay off in tremendous ways ten years down the road.

• We also need to trust the hardware. It's interesting that multi-processors have never really made the big time, isn't it? And I'll tell you why: it's because by the time you rework the operating system to support the new hardware cache coherency model, and you've finally re-designed and re-debugged the interrupt handling, it's 18 months later and the single CPU model is now twice as fast and selling for half as much, and you've got no way to recoup the cost of all that software engineering. Moore's law tells us that the number of transistors per processor will double every 12-18 months. That law has held true for a long time and shows no signs of being repealed now. The hardware is going to get faster -- you can depend on that, if for no other reason than there is so much money to be made by creating faster hardware.

  If you code for simplicity, you get your product out the door faster, you spend less time debugging, and it will run as fast as it needs to run.

• That said, make it as simple as possible, but no simpler. There are some projects that require tricky stuff and don't modularize well, e.g., a new virtual memory subsystem. The trick here is to realize that you've got a project that won't fall to the usual methods of attack, and design it into the schedule so that you can still get it done. Find a good engineer, or small team that are geographically close and that have a block of time to commit.

  It's easy to find projects where this didn't happen: they're the projects that are perennial failures: once a year some subgroup in the virtual team tries it and gets nowhere, or makes a little progress but get bogged down, or, actually get it "almost" done but have to mothball it because there's a release coming up and they can't quite push it out the door.

  Part of this bullet item is recognizing your strengths. You have a lot of man-power, but it's intermittent man-power, and it's geographically distributed. What are we good at? Writing Perl modules, internationalizing software, building inherently modular software like the C library and most of the UNIX kernel. What are we bad at? Writing a new kernel virtual memory. Writing database software, or other places where algorithms and a long software debugging and maintenance cycle does matter.

• There's a law of programming, Brooks' Law, that's also worth considering here. Brooks' Law states that the complexity and communication costs of a project rise with the square of the number of developers, while the work done rises only linearly. That Law has also been stated more simply as, "Adding programmers to a late project just makes it later." If you consider the built-in complexity and communication costs of virtual teams, I think that Brooks' Law understates the problem in the Open Source world.

  Manpower, the Open Source project's greatest resource, has a fatal flaw. It doesn't scale. In your groups, you need to concentrate on ways to turn what you're bad at into what you're good at. Have two of your best engineers create the framework and everybody else fills it in. You have a million device drivers? Invest your good people's time creating better abstractions for device drivers, not in writing the device drivers themselves.

  Finally, do everything that you can in a scripting language -- your programmers will be far more productive that way, and scripting languages are fast enough now that we need to start the conversion, at *least* enough that we're driving the design and evolution of the scripting languages to be sufficient for our needs.

Engineering doesn't matter.

• Now, in the same area of overall design goals for your software, I'd like to talk about engineering itself.

• For me, as an engineer, one of the uncomfortable truths is that the technology doesn't really matter. I hate that fact. It annoys me in the depths of my soul. But, I submit to you that if there's anything that we should have learned from Microsoft and Windows over the past two decades, it's that the technology barrier to the marketplace is lower than we'd have believed possible. Surprisingly enough, it turns out that people don't care as long as the technology mostly works and they can get their jobs done most of the time.

• I almost entitled this part of the talk "Don't Focus on Technology". Unfortunately, for most of us in this room, that's the *fun* part of the job. But the bottom line in the marketplace is that once the technology is "good enough" then any engineering work that you do after that time is fun, interesting, and good for the soul, but it won't matter a bit in the marketplace. And, I want to make it clear that I'm talking about the marketplace of ideas here, too, not only the jingle-jangle of the money. "Good enough" is the most important engineering number, and you ignore it at your peril.

  For example, Linux and BSD systems are making inroads in the Windows/NT and Solaris server market. Now, this is something I know something about: Sleepycat Software makes its money from high-performance, heavily-threaded database engines on 4, 8, 16, 32 processor systems. Applications trying to move as much data as possible. And Solaris is better than Windows at both base and threaded multi-processor performance. Regardless, Windows is penetrating the Solaris market. And, both Solaris and Windows are better than BSD and Linux systems in my type of high-end server applications. And it doesn't matter: both BSD and Linux are penetrating those markets.

• That's not to say that you shouldn't be spending your time on making your engineering better than "good enough". As I said, that's the fun part, and it's why most of us do what we do. But it's important to remember when you write your group's business plan, when you plan your release engineering, when you decide what features your software needs, that "good enough" engineering with good marketing will kill "great engineering" that's slower to market.

  And, the statement "engineering doesn't matter" leads to the corollary: "marketing is everything". As you consider your Open Source team, as you write your group's plan, you have to focus on who your target users are and how you're going to get them to try your ideas. You also have to think about who your target developers are, and how to get them to join your group.

• If you're here, you're probably a techie. We think of the world in terms of engineering truth, beauty and justice, with the unstated assumption that technical elegance and superiority and will prevail. Unfortunately, this is not how the world works. But there's a fundamental problem in your organization: most of the people do it because they like to do hard, serious, *good* engineering. But, that's only 25% of the problem, and so by definition your organization is a situation where you make cars for a living, and 75% of your people are there because they like building engines.

• Now, I'm making some assumptions about your mission statement. I'm assuming that your group wants your software to be widely used, and that you measure your success by how many people buy or use your work. If that's not the case, that's going to change how much of this talk directly applies to you.

• OK, OK, you say, so we're good to go. We've got a mission statement, we've got a plan, we've got more process than you can shake a stick at. In short, we're focused. What should we focus on?

  First, I'd like to tell you all how impressed I am with how the BSD groups are executing in general. Most of the things that I've talked about are done better, I think, by the BSD groups than by their competitors. That said, I'd like to talk very briefly about some of the specific issues that I think Open Source organizations and groups using Open Source software need to focus on.

Proprietary vs. Open Source

• First, proprietary vs. Open Source. I don't really want to talk too much about proprietary development models vs. Open Source development models.

  Most of the folks in this room are well-informed and fully convinced one way or another, and I don't think that I'm going to change anyone's mind. However, there are a couple of points that I've already made that I think reflect on the larger question of which developmental model is appropriate for any single project. However, as more companies consider what parts of their business should be Open Source, and more Open Source projects consider how to get revenue from their work, it's an important issue.

• First, let me give you the standard, canned answer many people give on this topic:

  The quality of the source code written by small, well-focused, well-managed, group of programmers working in a single location is generally better than that written by a large, amorphous group of geographically distributed programmers. You can have quality code come from Open Source projects or proprietary vendors: the question becomes, in which environment can you create the right group to write the code?

  In general, it's easier to create a small, well-focused group in a proprietary environment, mostly because you get them 8 hours a day, and you get to be firm when telling them what they're going to focus on. In particular, that means that you get them to do the boring stuff, for example, documentation and testing.

• Now whether or not you believe those statements, I submit that it just doesn't matter. Remember the rule of "good enough"? Quality is *not* the determining issue of what software wins. Once the essential set of features are present, it doesn't matter that your system is more stable, or that your SCSI driver has 10% more throughput. It doesn't matter that your network stack is cleaner and twice as fast over bursty high-latency links. The problem is that we're engineers, and we defend our code quality to the death. We're seduced by the fact that our code is "better" into believing that we'll win in the marketplace, and that is wrong.

• I spent a couple of years in the mind-set of "quality is what matters". But Linus Torvalds' comment, "Given enough eyeballs, all bugs are shallow" applies here. If it makes a difference, the Open Source code will be fixed. Someone, somewhere, out there, will fix the problem. If it didn't get fixed, then it never really mattered.

• If it's not obvious, I have no objection to the creation of proprietary software, myself. But in the current software environment, I believe that software companies need to select their targets with great care. They need to focus on specific add-on markets, where their technical support and advice are at *least* as important as the software itself. And there's nothing wrong with that -- Oracle, for example, makes more than half of its money from consulting, database tuning, training and so on. I think that the results are in: if you take a traditional software company model up against an Open Source development team, you're never going to do well, and you're likely to fail.

The OS is the last war.

• As a group, we're falling into the apocryphal trap of all military organizations -- we're fighting the last war, the war of the desktop. We spend lots and lots of time tweaking the systems infrastructure, arguing about what it should look like and bemoaning the fact that Word doesn't run there. To paraphrase Rob Pike, in a world where Grandma (let alone Mom and Dad) is online, the desktop is almost uninteresting -- our industry is designing systems and services for ordinary people, and the focus is on applications and devices, not on infrastructure and architecture.

• I think that we've reached the "steering wheel" point in the operating systems arena. By that, I mean the point where we know what services the operating system should provide, and those services aren't going to change a lot, if for no other reason than so much software has been written that depends on the current interfaces. Any new approach is going to spend most of its time writing compatibility layers to match standard interfaces. While it may be true that a joystick would be a better driving interface than the steering wheel for a car, by-and-large that war is over and it's not worth fighting it again.

• We need to shift the way we think about what matters with regard to usability and Open Source in today's world, not the world of 1985. We all know that the web is central to today's computing and development platform, but the trick is acting like we know that, and focusing our development in the appropriate areas.

• I'm hesitant, for obvious reasons, to try and tell you at what technical areas you should be looking. My record at predicting the technical future is abysmal, only about twice as good as most of the industry pundits.

  The only point of which I'm absolutely convinced is that we should expect the future to be wildly different from the past. I've been typing the command "ls" for roughly 20 years, now, and I think that should be enough for one lifetime. We *are* going to see more powerful, more intuitive desktop models. The Web took everybody by surprise, and I don't see any reason to believe that we'll see the next set of changes coming, either. That said, I think that if we keep our heads about us, if we create focused development groups with structure that will allow them to thrive, those new applications will be developed on BSD systems.

• There are a couple of major areas where we could gain an advantage. To quote Rob Pike, "System administration remains a deeply difficult problem. Unglamorous, sure, but there's plenty of room to make a huge, even commercial, contribution." At this point, I'm almost resigned to the fact that system administration is never going to be a solved problem on UNIX. That system administration will become less relevant over time only because there are fewer issues as hardware interfaces become fewer and relatively identical, and, there will be less reason for me to use UNIX commands as the desktop interfaces and program suites become more powerful and self-directed..

  I have always thought that one of the obvious choices for an Open Source group would be system administration, because I believe that it's a problem best attacked by creating a good framework and then having a large number of people contribute within that framework -- and, as Open Source, once available it should not be difficult to convince other Open Source maintainers to create their new application releases in the context of that framework.

• I also think there is other low-hanging fruit that we are uniquely qualified to pick. I mentioned system administration, let me add testing. Why don't we have a framework where people can easily write utility tests with slick graphic front-ends? Why aren't we writing those tests for our utilities and our applications? This is another area where we have the manpower to get the job done, because there are enough people that if everyone takes one utility we'll be done by Christmas.

• To wrap up, I think we need to be more daring and open to new ideas outside of our historic focus on the operating system. UNIX and Open Source have given us a worldwide set of common interfaces, and we need to take advantage of that. The BSD groups are falling behind in some ways because we're not riding the edge of the technology curve. Why don't we have good thread support? Why don't we have good Java, and particularly, Javascript, support? In order to avoid being marginalized as a development platform, we need to be the first to offer programmers the tools they need.

• We need to be bold about choosing projects and willing to tackle the impossible. The only reason a software project is impossible for us is because we haven't planned the framework that will let us divide the work up. I really believe that we have the biggest, best group of engineers that have ever been assembled in a single software project. What we need to do is leverage this tremendous collection of talent.

  Thank you for listening.