Source code on Github
- (In particular, see the fcrepo4 project)
Task List on JIRA
- Google Hangout for daily standups
- https://groups.google.com/forum/#!forum/ff-internal (private)
Java API documentation
CI/Code Quality/Continuous deployment
Google Docs Folders
Join the IRC Channel
- #fcrepo on irc.freenode.net (or: via Freenode Web IRC )
|Java 8||Fedora 4 is built and tested using Java 8, and uses some Java 8+ features|
|Maven 3||The Fedora 4 build uses Maven.|
|Git||The Fedora repositories are on Github|
Fedora uses git, hosted on GitHub, for version control. You can find the main repository at http://github.com/fcrepo4/fcrepo4.
Setting up your IDE
- m2e Connector
- EGit or JGit plugin
- Sonar plugin - config tbd.
- Set your Eclipse preferences, see fcrepo4/src/site/eclipse/README.md
There is a style guide, which should prevent some extravagant commits full of formatting changes.
Build - Maven
mvn clean verify"to run tests, "
mvn clean install"to build and install in local Maven repository (where it can be found by other local projects).
- When adding new classes, they should probably not go in the kernel. Look for a submodule or separate project that's appropriate for the functionality you're adding. Ask if it's not clear where something should go.
- fcrepo-http-api (https://github.com/fcrepo4/fcrepo4/tree/master/fcrepo-http-api) is a good example to follow when setting up new modules (though new modules should be in a separate repo, not in fcrepo4).
- Logging output can be configured using System properties. In general, stacktraces, RDF graph dumps and other verbose output should be logged at DEBUG or higher log level, even in tests. WARN and ERROR should be reserved for unhandled exceptions, failures to persist data, and other repository errors (not client errors that have been successfully handled and reported to the client).
The code style conventions used in the project are based on the style-guide defined of Fedora-3. They should prevent commits cluttered with format changes.
Here are the major rules:
- Four space indents for Java, and 2-space indents for XML. NO TABS
K&R style braces
- Do not use wildcard imports
- Write Javadocs for public methods and classes. Keep it short and to the point
- Avoid public instance variables; use accessors
- Use public methods sparingly; implementation details are not public
- Maximum length of lines is 120 characters.
Create Javadocs for types of at least the following descriptivity
Each source file should contain a license header much like the following:
Use the maven-license-plugin to check for and add missing headers:
IDE settings are located in the project source. IDE users are strongly recommended to apply these formatting settings.
- Eclipse settings are here: fcrepo4/src/site/eclipse
- IntelliJ settings TBD (purportedly, this plugin will let you use the Eclipse settings above within IntelliJ)
We're in the process of adding checkstyle enforcement to our modules (meaning, if you violate some of the major style rules, the build will fail).
To check for violations, run the following command:
Ensuring that the quality of the code base is maintained (as it is extended, refactored, or otherwise modified) can be addressed through the extensive coverage of functionality by unit or integration testing. One approach often undertaken within the domain of software development is that of test driven development, in which the appropriate testing suites outlining the behavior of some method are structured concurrently or before the method itself is actually implemented. Adhering to this methodology is by no means mandatory, but it does often assist in ensuring that the code base is quality assured to a higher degree.
Each module has unit tests in the
src/test/java/org/fcrepo/ directory and integration tests in the
The unit tests often use mock objects to create surrogates for the Modeshape and webapp machinery that much of the Fedora code interacts with. Some good examples of mock object usage are in the
fcrepo-http-api module (e.g., FedoraNodesTest.java) and in the
fcrepo-jms-indexer-core module (e.g. IndexerGroupTest.java). By convention, unit test classes are named as: <FunctionalClass>Test.java
Integration tests use embedded Modeshape, servlet engine, etc. to run real requests against the code. The web-based tests typically use HttpClient to make requests. Non-web tests, use injection to have the repository and service objects made accessible to the tests (the embedded repository and injection are configured in the
src/test/resources directory). Some good examples are in the
fcrepo-http-api module (e.g. FedoraNodesIT.java) and
fcrepo-kernel module (e.g., FedoraResourceImplIT.java). By convention, integration test classes are named as: <FunctionalClass>IT.java
Remember to follow the Code Style Guide when writing your test classes. This is especially true if writing a test class is your first foray into contributing to Fedora.
Running the tests
See the fcrepo4 README file for more detailed instructions concerning Maven settings, etc., to build from source.
The following page is a slightly modified version of the ModeShape workflow.
The Fedora project uses Git, GitHub, and pull-requests for our development. Basically, this process is as follows:
- Setup your GitHub account, fork our GitHub repository, and create a local git repository - This is done just once!
- Work an issue
Why do we like this approach?
This workflow has worked really well for us, for a couple of reasons:
- GitHub accounts are free, and you use your account for all projects you work on.
- GitHub pull-requests are essentially changes to Fedora code that committers/contributors are requesting be merged. GitHub makes it very easy for everyone to see/discuss/review proposed changes. If need be, you can alter your pull-request. If all is well, we will merge your changes into the codebase.
- People can work independently without a lot of coordination.
- You can collaborate with other developers using your public fork.
- Git makes a distinction between the author (you) and committer (a few Fedora committers). So you are recorded as the author of the changes.
How about good ol' patches? We still accept them, so if you prefer using the patch approach you can are welcome to create and submit patches.
- Sign CLA
There are a couple of steps to set things up the first time:
Step 1: Create a free GitHub account if you don't already have one. Be sure to set up your account with a SSH key pair and your email address(es) so Git can identify which commits are yours.
Step 2: Go to the Fedora repository on GitHub and click on the "Fork" button
Step 3: Clone your fork, using the private URL. At a command line, go to a directory where you want the new local repository, and issue the following:
When this finishes you will have a local clone of your fork, complete with a remote named 'origin' that points back to your fork on GitHub (not the original).
Step 4: Tell your local clone about the official upstream repository on GitHub:
This uses the public URL for the upstream repository, which is read-only. This helps ensure that your changes only go into your fork repository.
Now we've set up a local Git repository, we can talk about the steps you'll do much more often.
Pulling Upstream Changes
Start by ensuring that you are on the 'master' branch and that you have no local changes:
The last command should report something like:
Now, we need to pull any changes made to the official upstream repository due to new features or bug fixes. You can pull them into your fork two ways:
The first is more direct, but it only gets the changes for one branch ("master" in this case) and because the merge is not explicit it can be confusing for new Git users that the 'pull' actually results in a merge conflict (e.g., if the files you have changed locally were also modified in the upstream). The latter uses an explicit merge and also fetches all branches (including any new branches that were created since you cloned your repository and/or added 'upstream' as a remote). You can list all branches with:
If you do not see a branch in 'remotes/upstream' that you know is in the official repository, run the "git fetch upstream" and "git merge upstream/<missingBranch>" form.
Any local changes you have will be merged, and any local commits already in the upstream will be handled as a fast-forward merge (leaving your branch at the same commit as the "upstream/master").
Now that your local Git repository has the latest, go ahead and push all the new commits to your fork:
This is an optional step. The "master" branch on your fork is not really used at all, but you can keep it up-to-date with the upstream repository if you want.
All changes should be made on topic branches, typically named according to the JIRA issue. (Recommended naming convention "fcrepo-xxxx".) There is nothing special about a "topic" branch -- it is just a normal Git branch that you create and use for a specific topic (i.e., JIRA issue).
NOTE: Pull-requests that use the 'master' branch will not be accepted, even in your fork repository. There are too many things that can go wrong. Firstly, doing so will make it difficult to work on more than one (independent) set of changes at a time; working on 'master' will make your second set of changes dependent upon the first set of changes. Secondly, GitHub pull-requests are tied to branches, so GitHub will want to include all your commits into the same pull-request. Thirdly (and perhaps most importantly), if your commits are not approved, your 'master' branch history will have commits that don't actually appear in the official 'master' branch, and this could be problematic in the future. The 'master' branch in your fork should really be just the local branch that represents the official repository's 'master' branch - use it to pull changes from upstream and merge/rebase onto your topic branches.
To create a topic branch, start on a branch you want to base from (which is often 'master'). The following command creates a new branch named "fcrepo-1234" (based off of 'master') and then checks out that branch:
Work directly in the new "fcrepo-1234" branch. This is where you make your changes and run your new/modified unit tests. When you are happy, stage your changes with:
Do a complete integration build to make sure your new tests pass and that your changes did not cause a regression:
If you need to make more changes, be sure to stage them and re-build.
Once everything builds, you can then commit your changes to this branch. There are various ways to commit, but this form will commit those changes you've staged and launches the editor where you can type out your commit message:
Be sure to use an appropriate comment. The first line should offer a brief description of the change (less than 50 characters), followed by an empty line, followed by a more detailed description on one or more lines (each line not to exceed 72 characters), followed by an empty line, followed by "Resolves: <ticket URL>"
If you think it makes sense to commit multiple times on your topic branch, then feel free to do so. Having multiple commits on a topic branch is perfectly fine for large changes. However, if your topic branch only contains a small number of changes (e.g., fixing a bug in one class and then adding or changing a test case), then it is preferred that they all be made in a single commit.
Note: After a pull-request has been submitted (see below), subsequent commits (based on response to code review comments) should be pushed to the same development branch so that they will automatically be added to the pull-request. Do not squash or amend subsequent commits into your original pull-request commits because it makes finding the deltas much more difficult for the code reviewer.
If you have been working on this branch for a while, other changes may have been merged (from other pull-requests) into the upstream repository. Often times this is okay. However, sometimes your local change will be affected by recent merges. It is best to make sure that you update your local clone before you create your pull-request.
To do this, switch to the "master" branch, have Git obtain all recent commits, and then update your branch:
At this point, your "fcrepo-1234" branch has been updated (rebased), and you can proceed.
NOTE: Do not ever merge these topic branches onto other official branches, even on your fork repository. If you do that, your "master" branch (or any other official branch) will no longer reflect the official repository.
After you have committed changes locally (and pulled from upstream), you can commit your changes to your fork repository and generate a pull-request. Pushing the changes is easy:
This will push the commits on 'fcrepo-1234' up to the 'origin' repository (which is your fork on GitHub).
After pushing your changes into your GitHub fork, you have published your changes but have not really told the Fedora community about them. To do this, generate a pull-request on GitHub. If your commits are on a branch other than 'master', be sure to update the commit range (changing 'master' to the correct branch name). Then record this on the JIRA ticket, and use the URL to the pull-request and include a good comment that describes your changes. Finish the JIRA ticket; the ticket will then be ready to deliver, and the integration managers will be alerted that you have a pull request outstanding.
NOTE: If you use good commit descriptions and fill out a good pull-request description, then you can just paste the same description as the JIRA comment (without the summary line).
After the ticket is put "In Review", one of Fedora's integration managers will be notified of your new pull-request. The role of an integration manager is to review the incoming pull-requests and decide whether they should be accepted and on which branch(es) they should be merged. As such, only a few people have this responsibility.
An integration manager will review your request within a few days, and often much faster. They will comment on your pull-request via the discussion or line notes, or in the JIRA ticket. If they like what they see, they will merge your proposed changes into the correct branch(es), then "Close" your JIRA ticket. However, they may like to see additional changes, in which case they will describe in the pull-request discussion area (or in line notes, or JIRA ticket) what they would like you to change, and "Reject" the ticket, to indicate to the ticket owner that the ball is back in their court. If you disagree, just use the discussion area. Otherwise, go back to your local topic branch for this issue, make the requested changes, commit them locally, and push them to same branch in your fork repository (using the same commands as before). As long as you are on the same branch (not simply named the same, but actually the same branch), GitHub will automatically update the pull-request with your additional commit(s) and notify the integration managers again. Finish the JIRA ticket again. Once the changes are accepted, the integration managers will merge your commits into the upstream repository.
NOTE: After your initial commit, please do not perform "git push --force" on your branch. Doing so requires a complete re-review of the entire pull-request since it is not clear what all changes have been forced. After the code review is complete and ready to be merged into the master branch, you may indicate if you want certain commits to be squashed or not. Typically, if all of the commits are simply iterations on a single unit of work, your commits will be squashed by the integration manager before being merged into master.
There is actually nothing else you need to do. However, you may want to periodically clean up the topic branches that are no longer needed. (Note that if you want to make additional changes on a topic branch and have them go into the original pull-request, do not delete the topic branch in your local repository or your fork repository on GitHub.)
This command will delete a local topic branch:
This command deletes the remote branch in your fork repository on GitHub:
At first blush, the syntax of this second command is a little strange. It is actually a form of the "git push <remote> <localBranch>:<remoteBranch>" command. If you do not specify the local branch, this basically means push nothing onto the remote branch, and Git removes the remote branch.
Configure for Performing Code Reviews
Since pull-requests are used when offering patches for code review, if you are performing a code review, this is one way that you can configuration Git to simplify the process.
At the top-level of the pertinent project directory, change your ".git/config" file as follows (notice addition of line:5)
Refresh your local cache
Checkout the pull-request branch for review
- Enjoy the review
Process for using JIRA
- The REPORTER of a story is responsible for providing criteria for CLOSing a story. For code implementation stories, this includes provision of a test.
- Once the story has been evaluatued to be worked, it is moved from the RECEIVED state to the OPEN state.
- The ASSIGNEE of a story STARTs WORK on the story.
- The ASSIGNEE of a story STARTs REVIEW of the story when the work is completed.
- When putting the ticket into REVIEW, a link to the GitHub pull-request should also be included in the ticket
- Note, the ASSIGNEE should STOP WORK on a story if they are temporarily changing focus to a different task.
- The REPORTER (and/or the Tech Lead) reviews/tests the finished code. Then either CLOSEs or REOPENs the story depending on the outcome of the review.
- Once a story is CLOSED, the Tech Lead (or delegate thereof) merges the code into the codebase.
- Jonathan requests the Glacier Mock story. It goes into the RECEIVED state. It has no assignee. It has no acceptance criteria.
- At the sprint planning meeting, we assign acceptance criteria and move the story to the OPEN statue. The story has no assignee to do the work.
- If the story goes into the current sprint, it may not have an assignee (Don't assign assignee to stories outside sprints!).
- At the next daily scrum, Chris decides to work on this story. He STARTs WORK on the story, and makes himself the assignee.
- 3 hours later, Chris is done with the work, makes a pull request, and moves the ticket to IN REVIEW.
- Jonathan (the reporter) reviews the pull request (runs tests, etc.). When Jonathan thinks the ticket is complete, he accepts the pull request and moves the ticket to CLOSED.
- If the work is subsequently revealed to be incomplete, the Tech Lead REOPENs the story.
- Stories that have been accepted are moved to the Backlog.
- Stories that have been started must be moved to a sprint. Stories are finished and delivered in sprints.
Process for implementing a fix or feature
- Start work on the ticket in JIRA
- Create an issue branch in your local git
- When you believe the ticket is complete, push the issue branch to github
- Send a pull request to master linking the JIRA ticket in the description
- Link the JIRA ticket to the pull request or commit
- Link the pull request to the JIRA ticket
- Move the JIRA ticket to IN REVIEW
- The reporter should review the change and, if it appears to be complete (including at minimum Integration and Unit Test coverage) move the ticket into CLOSED.
- The reporter should delete the issue branch in github
- The PM or Tech Lead should evaluate the demo and Accept/Reject the ticket
6 VMs courtesy of Oxford
- Processor: 2 x AMD Opteron 6168 @ 1.90GHz (4 Cores)
- Motherboard: Intel 440BX, Chipset: Intel 440BX/ZX/DX
- Memory: 4096MB
- Disk: 107GB Virtual disk
- Graphics: VMware SVGA II
- Network: Intel 82545EM Gigabit
- OS: Ubuntu 12.04
- Kernel: 3.2.0-34-generic (x86_64)
- Display Driver: vmware
- Compiler: GCC 4.6
- File-System: ext4
- System Layer: VMware
Our github repository has a post-commit hook that trigger a build on ci.fcrepo.org. This build publishes maven artifacts to sonatype, and pushes the fcrepo-kitchen-sink war to futures6: http://fcrepo4.fcrepo.org/fcrepo/rest/.
Fedora 3 is also deployed on futures6: http://fcrepo4.fcrepo.org/fedora.
Remote debugging is very useful for tracing through the source code during the execution of the application. In order to enable remote debugging, you need to provide specific JVM options that tell the servlet container (Tomcat or Jetty) to open a port to which a debugger will attach. Additionally, you need to create a debugging profile in your IDE (Intellij or Eclipse) that specifies the host and port to which to attach the debugger.
On Ubuntu, update your /etc/defaults/tomcat7 file with:
Restart Tomcat for the options to take effect.
Jetty (with Maven)
Using `mvn jetty:run`, you can pass in the JVM options at the command line:
IDE Debugger Profiles
- Select `Edit Configuration`
- Create `Remote` configuration
- Change port to match what you passed above in your JVM options
- Run the debugger
- Select Run -> Debug Configuration
- Select "Remote Java Application" -> New
- Change Host and Port to match your settings and select "Apply"
- Change Host and Port to match your settings and select "Apply"
- Click "Debug" to run the debugger.
- JCR 2.0 Specification (http://www.day.com/specs/jcr/2.0)
- JCR 2.0 JavaDocs (http://www.day.com/maven/jsr170/javadocs/jcr-2.0/index.html)
- JAX-RS 2.0 Specification (http://download.oracle.com/otndocs/jcp/jaxrs-2_0-pfd-spec/index.html)
- Implemented by Jersey (https://jersey.java.net/)
- Dependency Injection:
JAX-RS 1.1 Specification (https://jsr311.java.net/nonav/releases/1.1/spec/spec.html)
A "distributed, hierarchical, transactional, and consistent data store with support for queries, full-text search, events, versioning, references, and flexible and dynamic schemas", supports the JCR API in addition to its own REST API.
- Project site (https://www.jboss.org/modeshape)
- Java Docs (http://docs.jboss.org/modeshape/4.0.0.Final/api/index.html)
- ModeShape Notes
An "extremely scalable, highly available key/value NoSQL datastore and distributed data grid platform", default storage for ModeShape.
- Project site (http://www.jboss.org/infinispan)