Current Situation
Where does each project publish?
Project |
Snapshots (groupId) |
Third Party Libraries (groupId) |
Releases (groupId) |
|---|---|---|---|
Akubra |
fc-snapshots (org.akubraproject) |
- |
fc-releases (org.akubraproject) |
DSpace |
dspace-snapshots (org.dspace) [4] |
dspace-repo (org.dspace) [4] |
dspace-repo (org.dspace) [4] |
DuraCloud |
- |
fc-thirdparty (org.duracloud) |
- |
Fedora |
- |
fc-thirdparty (org.fcrepo) |
fc-releases (org.fcrepo) |
Mulgara |
- |
- |
- |
Topaz |
topaz-repo (org.topazproject) |
topaz-repo ([2]) |
topaz-repo (org.topazproject) |
[1] Currently, libraries that are "third party" to the FCRepo project are split between the fc-thirdparty repository (for libs authored by projects outside DuraSpace) and the fc-releases repository (for libs authored by other DuraSpace projects). The current FCRepo groupId/artifactId naming convention is further documented here.
[2] The Topaz project has historically put all third-party libs under separate groupIds within their repository.
[4] DSpace release artifacts are not exposed via local dspace-repo, all release builds use the central repo at http://repo1.maven.org/maven2/org/dspace/ unless the developer configures maven to behave differently. All third party requirements not already in the central repo or other popular repository such as java.net or codehaus (excluding oracle JDBC drivers) are published under the org.dspace groupId so they will be properly rynced to the central repository after release.
Repository details
Repository |
Hosted At |
Browsable? |
Write Access? |
GroupIds Published to Central |
Repository Platform |
Deployment |
|---|---|---|---|---|---|---|
OSU OSL |
Yes |
Manually granted by mdiggory |
org.dspace , org.duraspace [3], org.fedora-commons [3] |
Apache HTTPD Filesystem, WEB-DAV can be made available |
SCP Wagon |
|
OSU OSL |
Yes |
" |
None |
" |
" |
|
Cornell |
Yes |
Manually granted by cwilper |
None |
|
|
|
Cornell |
Yes |
" |
None |
|
|
|
Cornell |
Yes |
" |
None |
|
|
|
PLoS |
Yes |
Manually granted by Ronald? |
None |
|
|
[3] These groupIds are not used. Generally, groupIds of projects within DuraSpace are tied to the project, not the overarching organization. Also, the Fedora Repository project has now standardized on using org.fcrepo as its groupId. (MRD: I agree)
What Repository Platform might be utilized
Currently OSL is working on implementing java based services for deploying webapplication driven services. At this time they are not announcing any timeline for this.
What Merged Repositories Might Look Like
Repository |
Hosted At |
Browsable? |
Write Access? |
Published to Central? |
|---|---|---|---|---|
duraspace-repo |
OSU OSL |
Yes |
Granted by admin(s) for each project, under that project's groupId |
Yes |
duraspace-snapshots |
OSU OSL |
Yes |
" |
No |
Participating Projects:
Should:
- Comply with central requirements; all artifacts published by each project (third-party or not):
- Need to be under the groupId of the project responsible for authoring/publishing the POM.
- Need to be freely distributable.
- Designate admin(s) who can delegate write access to the portion of the repository under that project's control.
10 Comments
Mark Diggory
I think an excellent idea might be to consider how to put a combined repository into the AWS EBS based not just on the dspace/fedora/mulgara artifacts, but also many of the required artifacts for the maven build process. I was recently building a dspace instance in AWS and realized that I would significiantly reduce my incoming bandwidth usage if I had a full tree of the DSpace Maven dependencies present as an internal repository mounted via EBS, if mounted and an alternative settings.xml were used, the result would be pretty snazzy.
http://confluence.atlassian.com/display/BAMBOO/Populating your EBS volume
I'd almost suggest this would make a very good project for the whole of the maven community, a combined EBS volume that represented the entirety of the Maven Central Repository would eliminate any bandwidth usage during builds.
Chris Wilper
Hi Mark, congrats on the 1.6 release. I think Fedora and DSpace are both now at a relatively quiet point in our release cycles and I'd like to pick back up on this thread.
I've been looking at Sonatype's OSS hosting and wonder if that would fit the bill for most of our needs. The nice thing about it is that they're doing it for free and providing the sync up with central automatically...no sysadmin work for us.
I really like the idea of having a combined m2 repo on EBS...I recall something similar being done for the ubuntu package repositories...not sure if that's still the case. Ideally, a mirror of central. I wonder if this is something the Sonatype guys might be willing to set up to showcase their product...hmmm.
One thing I'm not sure about (haven't asked yet) is whether we could have multiple accounts allowed to deploy to our Sonatype-hosted repositories. The way it looks, I think the assumption is that you have one account capable of doing deployment for each groupId.
Mark Diggory
I say +1 on all these ideas... A good tool like Nexus will make our lives much easier. And free Hosting is always a plus. Can you open a ticket and ask them if they will allow us to host multiple groupIds under it? Other wise, I'd say to just manage each as a separate repo. I suspect they have encountered is and will give us the best recommendation they can.
One other question to ask is about DNS resolution, so we can resolve existing builds against the new repository. Otherwise, we will need to maintain httpd redirects for the existing repositories.
Mark
Chris Wilper
As you've found, my initial ticket is here:
https://issues.sonatype.org/browse/OSSRH-307
This was quite easy to get them to set up. It looks like regardless of whether projects are hosted in the same logical repository on Sonatype's end, the access controls are by groupId...either way, I think their model works fine with our scenario, since artifacts released by either project are done via distinct groupIds.
Also, looking at the Central Sync Up Requirements, we on the Fedora side have a bit of work to do before we are really central-friendly (dependencies on artifacts in other repos besides our own and central...).
Mark Diggory
TBH, we rely on artifacts in repos other than central... Sakai and java.net Its never limited putting stuff in central in our syncs. I wonder if Nexus is more explicit about testing this.
Chris Wilper
Yeah, I was wondering how lenient they actually are with that. Though I can understand the value to having a more strict standard. The java.net dependency sounds pretty benign, and may be worth looking at central to see if what you need is published there yet (I speak from experience, having just upped the version of JTA Akubra depends on after finding the latest version is now published in central).
I've submitted a ticket to get Akubra (groupId=org.akubraproject) hosted there too, and plan to push that to central soon, since its poms are closest and it's a dependent of org.fcrepo anyway. I've been impressed with the responsiveness in getting the initial set up so far, but will let you know how the end-to-end process goes with Akubra.
Chris Wilper
RE: Strictness with dependencies in central
It looks like the preferred process has changed, and they are indeed getting more strict with central. See Brian Fox's comments at the bottom of this post. Also, note the Approved Forges section of the Guide to uploading to Central.
RE: Experience using Sonatype's OSS hosting for Akubra
After they gave me access, I went ahead and did a snapshot release with no trouble. Then I got Akubra's poms in line with the standard and did a new version release. I got through the process ok and left a note in my Akubra Hosting Ticket yesterday. I suspect they'll do the first-time sync with central early this week after someone gets a chance to review it. For future Akubra releases, syncing to central should happen within an hour of promoting the staged repository.
I logged most of the gory details in the new Akubra Release Process document.
Mark Diggory
Forges are different than external repositories.
But... http://nexus.sonatype.org/oss-repository-hosting.html
"Make sure your POM does not contain repositories or pluginRepositories element. Central repository must be self-contained, so all your dependencies must be available in Central. You need to make sure your project can be built without extra repositories or pluginRepositories."
Sonatype seems stricter... I'm not sure what to do about this... we may need to talk with SAKAI about pulling Sakai dependencies under our own group ID. My understanding is that Sonatype is a process of absorbing the JAVA>NET repository.
Chris Wilper
I don't think the folks who manage central are going to disturb pre-established rsync anytime soon, but they've become more strict with newcomers. Sonatype's requirements for OSS hosting appear to mostly just extend from central's updated requirements for new projects (gpg keys, all dependencies in central, etc).
I suspect it will be a while before Fedora artifacts can be published to central under these rules; several of the artifacts we depend on are not wrapped by us, nor available in central.
Non-Central Repository
Dependencies Hosted
http://download.java.net/maven/2/
com.sun.jersey:jersey-bundle:jar:1.0.1
com.sun.woodstock.dependlibs:jhbasic:jar:2.0
gnu.getopt:java-getopt:jar:1.0.13
javax.transaction:jta:jar:1.0.1B
javax.xml:jsr173:jar:1.0
http://maven.ow2.org/maven2
flexlib:flexlib-bin:swc:2.4
http://repo.aduna-software.org/maven2/releases/
info.aduna.commons:*
org.openrdf.sesame:*
http://repository.jboss.com/maven2/
fast-md5:fast-md5:jar:2.5
javax.resource:connector-api:jar:1.5
http://repository.sonatype.org/content/groups/flexgroup/
com.adobe.flex.framework:*
Sigh.
Mark Diggory
http://www.sonatype.com/people/2010/02/java-net-maven-repository-rescue-mission-on-march-5th/