Utilizing Instability and Breakage to Promote DSpace Evolution, Code Review and Developer Participation

Proposal for less constraints on Trunk:

To Give DSpace Developers opportunities to work on the trunk without restrictions that everything going into it be completely perfected gives the developer community and the codebase opportunities to evolve. We recognize that sometime disturbance and breakage has a stimulating effect on developer participation.

Therefore it is proposed that the DSpace Guidelines for Committing to trunk allow for period of instability and stability with clear deadlines as to allow for the introduction and testing of new features which have been decided to be included into the next release by the community.

Commiter Rights and Google Summer of Code Participation

Background Discussion of GSoC Commit Rights:

This proposal originally arose out of the Google Summer of Code project with the need to actually have GSoC students help with the code merge of their projects into Trunk. But, as students are not "Committers", they currently are limited in receiving rights to Trunk (under our current policies). It is recommended that we hold a Special Topic meeting next week (July 28, 2010), and try and determine what to do in this scenario. Obviously, this proposal then leads to other questions of how we may view committer rights.

Discussion of how best to begin preparing/merging "ready" GSoC code in preparation for DSpace 1.7 began in the Developer Meeting on July 21, 2010.

Options for merging of GSoC code into Trunk for 1.7:

Four options were discussed (there may be more options available):

1. Require students to create JIRA issues for their projects? (Preferable to have several tickets for a single project, breaking it into components in varying states of readiness)
   • We'd probably want to encourage this, no matter which merging option we choose below [Tim Donohue]
2. Allow "ready" projects (or parts of projects) to be merged & committed immediately to Trunk (by the students themselves)
3. Allow "ready" projects (or parts of projects) to be merged & committed immediately to a Branch (by the students themselves). After a quick review process, this Branch would be merged back into Trunk for DSpace 1.7 release.
4. Keep the GSoC projects where they are (in SVN). Schedule them for review, and have a Committer (GSoC mentor?) merge into Trunk later on.
   • This option is how we've done things in the past. It is worth noting that, to date, no GSoC project code has ever been accepted into Trunk, as oftentimes they are "abandoned" or left in an uncertain state after the students complete their work.

Key Questions:

1. How liberal or conservative do we want to be with allowing GSoC students to commit/merge "ready" code in preparation for DSpace 1.7? This includes:
   1. How liberal/conservative do we want to be about giving students temporary commit rights to Trunk? Or, would we rather they merge their code together elsewhere (e.g. a common branch based on Trunk)?
   2. How liberal/conservative do we want to be about allowing for temporary "breakage" of trunk (which could happen as several projects attempt to merge code)?
2. How much extra reviewing do we want of GSoC projects whose Mentors feel the code is "ready" for broader distribution/release in DSpace 1.7? If extra reviewing is warranted, how do we want to ensure this review is done in a timely manner (i.e. in time for DSpace 1.7, as necessary)?

(Feel free to add your own comments to these questions directly on this wiki page.)

Issues In Relation to Asynchronous Release and Modularity

Robin Taylor says

Having completely forgotten about yesterdays developers meeting (doh !) I just read the log and was interested in the chat about committer rights etc. I don't want to preempt any further discussion but just wanted to say that it might be logical to have the discussion about modularisation and asynchronous releases before the discussion on committing to trunk. The envisaged shape of the svn repo might affect how we view committer rights eg if Sword lived in its own module with its own release cycle then people might be less concerned about an individual with an interest in Sword having commit rights just to that module.

Original Dialog In IRC Meeting

(Also see the meeting notes/transcript at DevMtg 2010-07-21)

[20:03] <tdonohue> mdiggory: are there any GSoC upcoming meetings to know about (I know this was mentioned as still being worked on at/before OR10)?
[20:03] <mdiggory> Monday, August 9 Suggested 'Pencils Down'
[20:04] <mdiggory> Friday, August 20 Final Evaluation Deadline/Final Payment Issued
[20:04] <mdiggory> Monday, August 23 Final Results Announced
[20:04] <mdiggory> Monday, August 30 Students Submit Code Samples
[20:04] <tdonohue> (FYI for all -- this schedule is also embedded on our GSoC wiki page: DSpace Summer of Code2
[20:04] <mdiggory> All students did receive passing grades for the midterm.
[20:05] <tdonohue> that's great news all around!
[20:05] <mdiggory> But, now I think begins the challenging period of us... how do we finish out this years to have the projects follow through to codebase / trunk where appropriate
[20:05] <mhwood> JIRA incident to track each one?
[20:07] <mdiggory> Well, we started out with some failure to setup a JIRA project for GSOc, now I'm feeling that was maybe a bad choice...
[20:08] <mdiggory> so, now that we have material to bring into various levels of core, it would be good to see tickets in appropriate places for those activities "DSpaceJIRA" , "Services JIRA" etc
[20:09] <tdonohue> yea -- it's better to split up into manageable "chunks"
[20:10] <mdiggory> But, simply getting everyone used to creating tickets around the GSOC projects is the first challenge
[20:10] <tdonohue> it'd be harder to "approve" a JIRA issue if it's really just a placeholder for the whole project -- I imagine for some projects only "part" may be trunk-ready, and the rest may need a bit more work
[20:10] <mdiggory> I see Unit testing as a good first candidate
[20:11] <tdonohue> well -- couldn't we consider the creation of final JIRA issues as part of the "final project report" (i.e. the final deliverable)
[20:11] <mhwood> Binding each project more tightly to its place in DSpace than to GSoC also emphasizes that this is "real", not just an exercise. We want to see this stuff become production-ready.
[20:12] <tdonohue> mhwood +1
[20:13] <mdiggory> I agree, that not all the projects will be good candidates for trunk, but ticketing for other projects needs to target appropriate modules projects
[20:14] <mdiggory> so for storage and semeantic web projects we have the "DSpace Services" JIRA project that might be a good home
[20:15] <tdonohue> mdiggory -- yes, I agree with that. That's what I was trying to get at by saying that a "JIRA placeholder" is less useful than splitting up the project and adding separate (but related) JIRA issues into each of the appropriate modules in JIRA
[20:15] <mhwood> Might even want several tickets for a single project if it usefully breaks down into components in varying states of readiness.
[20:16] <tdonohue> exactly, mhwood -- in my opinion, several tickets is much better than one ticket (as most of these projects touch several parts of the codebase)
[20:18] <mdiggory_> ok, I was going to say that the REST work, if we are truly targeting 1.
[20:18] <mhwood> Time to go fishing in syslog.
[20:18] <mdiggory_> 1.7 with it, might have a home in the dspace 1.x ticketing
[20:19] <tdonohue> yea -- I was hoping that both REST and some of the very early Unit Testing could start to be pushed towards Trunk (and DSpace 1.7)
[20:19] <mdiggory_> (notes this will open up a discussion into async release process etc)
[20:20] <tdonohue> (but whether my hopes match with reality is another matter -- in any case, JIRA tickets should help us all get a better sense of how "ready" each of these projects are)
[20:21] <kshepherd> morning all, sorry for lateness
[20:21] <tdonohue> hi kshepherd -- still talking GSoC right now
[20:21] <mdiggory_> I think a goal with the REST work is that it support legacy DSpace at the same time as new services portings... IMO it should be pluggable to that degree and I know that Aaron would probably recommend such a target.
[20:22] <mdiggory_> I think there is a point where some destabilization of trunk needs to happen to get the work in... and we need to aggree on that period as much as on the feature freeze dates etc
[20:23] <mdiggory_> sso, for instance, if destabilization happens in sept then feature freeze could happen in oct
[20:23] <mdiggory_> or august/sept
[20:23] <tdonohue> that all makes sense -- I still think the best first step is to have someone (preferrably the students, perhaps with mentor help) to create the appropriate JIRA tickets for each project -- that way we can push them for review and try and come to a quick consensus on next steps
[20:24] <mdiggory_> I agree, but we need to agree on who does that commit work, and I will suggest students over mentors
[20:25] <tdonohue> mdiggory_ you mean actually having the students commit the code to trunk? (or am i misunderstanding)
[20:25] <mdiggory_> yes
[20:26] <mdiggory_> We talked about temporary commit rights etc at the or10 meeting...
[20:26] <tdonohue> hmmm... as much as I'd like to say "yes", that does make me slightly nervous (in that we haven't had a more thorough review of the code by committers to help decide which parts are 'non-controversial')
[20:26] <mdiggory_> my point in suggesting it is to aleaviate yet another gatekeeper role / bottleneck in the process
[20:27] <tdonohue> yes, we did talk about the temporary commit rights at OR10 -- but if i recall, it was tabled for a full-meeting discussion (special topic mtg)
[20:27] <mhwood> Commit to a branch and then merge slightly later?
[20:27] <tdonohue> yea, i definitely see the benefits mdiggory -- i completely agree with minimizing bottleneck (just worried at whether this is controversial amongst our committers)
[20:28] <kshepherd> it makes me slightly nervous ;)
[20:28] <tdonohue> commit to a branch first may be less controversial
[20:28] <mdiggory_> I challenge that this community needs to change its perception of trunk.
[20:28] <kshepherd> on the other hand, i've argued that broken trunks aren't the end of the world
[20:28] <mdiggory_> but and do so respectfully, I know why the first blush is to be conservative.
[20:28] <tdonohue> mdiggory_ i understand your challenge (and honestly, I agree to a point) -- it's just something that needs to be voted on
[20:29] <mdiggory_> of course
[20:29] <tdonohue> so -- if you want to propose this (likely via email, we don't have enough here now), we can bring to a vote and see what happens
[20:29] <kshepherd> if we really can't live with broken trunks, we'll just keep our own branches on our own svn repos anyway
[20:30] <kshepherd> so i think i'd vote for it.. might need to think more though ;)
[20:30] <mdiggory_> mhwood: unless there is a review between the commit to the branch and the merge... I don't see usefullness in that strategy.
[20:30] <tdonohue> I see three options: (1) allow commit to trunk , (2) allow commit to a branch, and later merge to trunk (after more review/testing), (3) wait to move to trunk until review/testing (the old way of doing things -- which sometimes leaves things languishing)
[20:30] <mhwood> The branch is to create space for review, with the code checked into the SCM so we can use all the usual SCM tools for inspecting and manipulating it.
[20:31] <mdiggory_> thats already happening
[20:31] <mdiggory_> but we need to be very cautions about how long those branches sit... deviating from trunk
[20:31] <tdonohue> mdiggory_ it's happening separately right now though (each project has it's own 'branch') -- I think mhwood is suggesting a "merged" branch
[20:32] <mdiggory_> the longer they do so, the lower their probability for inclusion
[20:32] <mhwood> Agreed: *prompt* review and decisions.
[20:32] <tdonohue> agreed -- I think the goal would be that the branch is very temporary -- a place for immediate review and then hopefully move to trunk before 1.7
[20:32] <mdiggory_> tdonohue: I just think thats a not such a good idea, this is really what a trunk is for
[20:33] <mhwood> So where should we be looking? The SCM tree seems to have grown like kudzu lately.
[20:33] <mdiggory_> and I agree with kshepherd, breakage on the trunk is actually a good thing
[20:33] <mdiggory_> is the disturbance that promotes change/improvement
[20:33] <kshepherd> yeah we do need to get less precious about trunk, i think ;)
[20:33] <tdonohue> mdiggory_ yes -- but, do these students have experience with merging? I'm worried that a merge could accidentally break more than we bargained for (but I could just be paranoid)
[20:33] <kshepherd> and we have more code review tools to help with post-commit reviews, too
[20:34] <mdiggory_> tdonohue: that is why we are here as mentors, to assist them in learning such things if they do not
[20:34] <mdiggory_> ie mentoring = helping
[20:34] <mdiggory_> not mentoring = judging
[20:34] <grahamtriggs> depends on the way you want to view it - yes, we could get less precious and more experimental with trunk... but that would mean we should be branching early for releases
[20:35] <mhwood> Yes, be as experimental as we want, so long as there is an island of relative stability *somewhere*.
[20:35] <mdiggory_> grahamtriggs: or we work in phases/periods of stability /instability
[20:35] <mdiggory_> ie... controlled chaos
[20:36] <tdonohue> so, i'll also note here that "breaking trunk" currently is against our recently approved "Guidelines for Committing" -- Guidelines for Committing (obviously, we can change these guidelines as we see fit, but we need to have larger discussion)
[20:36] <mdiggory_> So, table for special topic meeting that needs to happen asap?
[20:36] <mdiggory_> so more developers that have an interst in such a policy have time to respond
[20:37] <tdonohue> mdiggory_ : yes, we could even schedule mtg for next week -- but, we need to also send an email proposing these changes (for those unable to attend)
[20:38] <mdiggory_> I volunteer kshepherd ;-)
[20:38] <tdonohue> mdiggory_ : will you take lead on proposing this to dspace-devel today/tomorrow? We can put on agenda for next week's mtg.
[20:38] <kshepherd> hmm? for what?
[20:38] <tdonohue> haha -- you are the one who brought this all up, mdiggory_ :)
[20:39] * kshepherd runs svn delete on trunk and runs away cackling
[20:39] <tdonohue> haha :)
[20:39] <mdiggory_> to write an email about proposing to change the Guidelines to allow more breakage or destabilization phases on trunk
[20:39] <mdiggory_> ok ok ok...
[20:40] <tdonohue> yes -- and explain the background as to *why you propose this* -- which is that the proposal is that the GSoC students commit ready code to trunk
[20:40] <tdonohue> those are two separate issues, I know -- but we might be able to cover both in one special topic mtg
[20:40] <kshepherd> my "break trunk" opinions have mostly been in the context of the way we tend to leave bugfixes going stale in JIRA instead of just committing.. this is a slightly different context so i'd like to read/think more
[20:42] <tdonohue> ok -- anything else immediately on GSoC? I notice we only have 20mins left (not that there's much more substantial on the agenda though)
[20:42] <mhwood> kshepherd, that is a bit different. I think many of us tend to want more comment than we give, so changes rot....
[20:42] * kshepherd nods
[20:43] <mdiggory_> but I think it is related
[20:43] <kshepherd> no better way to enourage peer review than accidentally breaking trunk, though ;)
[20:43] <mdiggory_> because it has to do with getting things into trunk, then hardening them when errors arise
[20:43] <kshepherd> so like mdiggory_ says, it can actually force us into action
[20:43] <tdonohue> true -- but we also don't want to harm productivity of our widely distributed group by having a broken trunk for too long :)
[20:43] <mdiggory_> rather than waiting long periods of time and only exceppting the most perfect merges
[20:43] <kshepherd> and will get us used to using atlassian code review tools more
[20:44] <mdiggory_> which raises the bar for contribution
[20:44] <mhwood> Do those tools work now? Must go look again.
[20:44] <kshepherd> tdonohue: heh true, of course.. bear in mind that i think 95% of commits won't actually result in a broken trunk ;)
[20:44] <tdonohue> yea agreed :)
[20:44] <kshepherd> mhwood: heh, i hope so! i think there is a test review in crucible
[20:44] <mdiggory_> yes, crucible is still working