Mar 13, 2016

MongoDB v 3.2 + silently breaks backward compatibility affecting behavior of capped collections

Capped collections are collection with some very special characteristics,  although these are used internally by MongoDB for replication (oplog) are also offered as a feature for general use, their limitations and  advantages are well documented.
Because of those advantages developers are using capped collections widely for many purposes, and this usage it is not a hack or undocumented feature exploitation because capped collections are well documented and a well known feature of mongoDB.
I have seen capped collections used as :

  1. FIFO buffers
  2. Trigger-like mechanisms
    (as a matter of fact a well known framework is based on this feature
  3. PubSub architectures
  4. Message queues
    Yes you can implement a message queue with mongoDB, although it will not be as fast as ZeroMQ or some other tools, it has some attractive benefits as persistence, built in redundancy, limiting stack complexity etc. (My implementation of a PubSub/message queue)  
Except that you can't delete a document in a capped collection the other major limitation used to be that a document couldn't grow up in size as a result of an update. This is well understood by the developers and could be addressed by various techniques as:
- pre-filing field(s) with dummy values
- un-setting a field during an update operation to make space for a new field(s) etc.

Now all of a sudden MongoDB from version  3.2.0-rc0 + decided to introduce one more limitation "the document can't shrink in size" as a result of an update.
This breaks backward compatibility of a well published and widely used feature and although I understand the technical reasons behind this decision as described here, still I can't accept that such a decision can be taken so lighthearted without consulting with the developers/ecosystem. If you follow the discussion in the above ticket in Jira you get the impression that the only thing that they really cared about is how this can possibly affect their internal use of the feature and since they found no side-effect they went ahead with this.
To keep adding salt to injury the breaking change is not yet published in the manual as of today, neither I can find any reference in change logs.
So it is left to developers to find it out the hard way when their code breaks after an update.
I filled a ticket, then I realized a related ticket already exists but there is no action taken yet.
Of course those tickets deal with the documentation only since the code breaking change is there and we have to live with it.
Too bad many developers didn't realize this change was planed, that's why a compatibility breaking change should get as much publicity as possible so that developers don't get caught of guard, since is impossible for a developer to follow each and every ticket in Jira.

This is a sad story that  I hope will not be repeated in future since some of the things that made mongoDB so successful IMO are:
a) developers are not caught up by surprises
b) high quality of manuals