Some idle thoughts on PostgreSQL project management

Replication is a topic of perpetual discussion in the community: how it is designed, how it is integrated into the back end, and whether a given approach is the right one for any given job. Every three or four months, the topic erupts on some list or other, invariably with a suggestion by someone that some approach is "obviously" the right way to do it.

These discussions always include the observation that replication is a big topic that has many applications, and the idea has always been to provide more than one facility, to satisfy the various kinds of applications. This is consistent with what several other DBMS do; and there is no evidence that a single system is on the horizon, which can do everything for everyone.

In 2006, frustrated by the poor results from these occasional discussions, I tried to set up a PG foundry project that would come to some agreement on the minimum changes needed to the main tree for various replication systems. I reasoned that, since there are several different projects, each attempting to deliver replication, we had enough experience to be able to draw a first pass at an API. Then we'd be able to come back to the community and see whether there was real interest in that API.

This approach has more or less completely failed: there were some early postings by a couple different projects, but most replication system designers rejected the list as a complete waste of time, and the idea of designing an API was dismissed as burdensome and complicated for the replication system designers.

I have concluded that this result was predictable, because of the ways features are developed in the community. This talk outlines the ways I think feature development actually happens, explores some of the costs and benefits of that development model, and tries to suggest some ways that larger, more complicated systems will be possibly developed in the face of the actual development strategies in use.

I first outline what the official approach is: come up with an idea, discuss in the community, find consensus, then implement.

I then discuss some kinds of strategies more often really seen.

The first is the single-source proposal that emulates the official approach. These are very often improvement proposals: "Here is a chunk of behaviour that is missing or wrong." This conforms nicely with the TODO-list system we have.

The second is a more complicated proposal: the code is already written, or nearly done. This approach is far more common that is usually acknowledged, and it sometimes leads to fractious discussion.

The third is a degenerate version of the second: some filthy hacks have been applied by someone, and they come with a proposal to fix up the hacks and get them integrated in the code.

The fourth is the frustrated patch: when consensus can't be achieved under the first approach, some developers go away and develop a patch that implements the proposal anyway, in the hopes that running code will be convincing.

Each of these has a number of costs to the community. None of them really allows for large community planning of multi-release feature development, which is why bigger features are sometimes hard to discuss.