*Deprecated* This material is for historical purposes only See https://wiki.duraspace.org/display/VIVODOC/All+Documentation for current documentation
*Deprecated* See https://wiki.duraspace.org/display/VIVODOC/All+Documentation for current documentation
This page describes the process of creating releases for VIVO.
Each step in the process is described, along with an example of the commands used to accomplish that step. If a script is available to accomplish the step, then the script is described and a sample run of the script is illustrated.
The release process includes:
The release process is more than just building the files. For the larger picture, check Release process
A test candidate is a packaged distribution that is known to be incomplete for the release, but that allows the testers to begin their work. These are optional, and some releases won't use them.
A code freeze is not issued for a test candidate, so a release branch is not created. Development continues on the develop
branch.
A tag is added to the develop
branch to show where the test candidate was created.
A release candidate is a packaged distribution for which there are no known impediments to release. The JIRA issue tracker should contain no blocking issues when the release candidate is created. If testing finds no substantive problems, then the release candidate will become the release.
A code freeze is issued prior to the first release candidate, and a release branch is created. Development on the develop
branch will not be included in the release. Bug fixes on the release branch should be merged into the develop
branch, so they will be present for subsequent releases.
The final release is just that. When a release candidate passes all tests without substantive issues, a final release is created.
In early releases of VIVO, the final release was just a copy of the last release candidate. Now, a release includes files of revision information, so the final release cannot be a mere copy, and must be built like the other release products.
When the final release is built, the release branch is merged into the master
branch, and the final tag is moved to the master
branch.
After the final release, additional changes may be made to the release branch, but they will not be used unless a bug-fix release is made.
The release consists of the files stored in the VIVO and Vitro repositories on GitHub, except that:
.git
directories are removed, so the release files can't be used to commit changes.Vitro
files are placed inside the VIVO files, in a directory named [vivo]/vitro-core
revisionInfo
files are added, one for VIVO and one for Vitro, stating the latest release name and commit hash.A collection of Ruby scripts has been written to help automate this procedure. Using the scripts means avoiding typing errors and reducing procedural errors. For example, you can be confident that the scripts will follow the VIVO naming conventions for branches and tags. The chance of typing mistakes is reduced to essentially nil. And automated checking of pre-conditions helps to insure that the steps are not performed out of sequence.
On the steps that have been implemented in Ruby, this page describes the effect of the script and shows a sample run.
The scripts are stored in the VIVO repository, under [vivo]/utilities/releaseScripts
. The directory containing the scripts may be copied to a useful location.
You should not run the scripts directly from the VIVO repository that is being packaged for release. The scripts themselves will modify that repository as they run, and this could lead to problems in execution.
Run the scripts by invoking the Ruby compiler. The scripts require at least Ruby 1.8. For example
ruby 1_setup.rb
The repositories that the release is built from must be kept clean. Any extraneous files that are created in the repositories may end up in the release.
Don't run ant
against the repositories. The ant script creates a large .build
directory, which should not be included in the release.
Don't open the repositories as projects in Eclipse. Eclipse creates an assortment of artifacts, including a large bin
directory, which should not be included in the release.
Before running the release scripts, you need a directory that holds copies of the VIVO repository and the Vitro repository. This directory will also hold the temporary files and the release files as you go through the release process.
In order to use the scripts, or to follow the commands on this page, the repositories must be named VIVO
and Vitro
.
To create the repositories in a directory called /usr/local/BuildingReleases
, execute these commands
mkdir /usr/local/BuildingReleases cd /usr/local/BuildingReleases git clone https://github.com/vivo-project/VIVO.git VIVO git clone https://github.com/vivo-project/Vitro.git Vitro
As you make changes to the repositories, creating branches and tags, git will record your name and email address on the changes. Make sure that they are set correctly.
To configure these settings, the script will execute these commands:
git config --global user.name [your name] git config --global user.email [your email address]
For example:
git config --global user.name 'Jim Blake' git config --global user.email 'jeb228@cornell.edu'
The script will execute similar commands to store additional parameters (see below).
This script will prompt for your name and email address, and set them in git.
It will also prompt you for values that will be used in subsequent scripts:
Finally, the script will suggest running the license checker, to insure that any recent changes have the proper license tags. This is not required, but it is recommended. When submitting fixes to the release candidates, people frequently overlook the licensing requirements.
engin:releaseScripts jeb228$ ruby 1_setup.rb Git base directory? (holds Vitro and VIVO repositories) (Currently not set) /usr/local/BuildingReleases Setting Git base directory to '/usr/local/BuildingReleases' Git user.name? (a string) (Currently 'j2blake') Keeping Git user.name as 'j2blake' Git user.email? (a string) (Currently 'jeb228@cornell.edu') Keeping Git user.email as 'jeb228@cornell.edu' Release label? (like '3.2' or '3.2.1') (Currently not set) 1.8 Setting Release label to '1.8' Release candidate label? (like 'rc1' or 'tc3' or 'final') (Currently not set) rc1 Setting Release candidate label to 'rc1' It's a good idea to check the licenses before proceeding. Ready to run the licenser? (y/n) y Scanning VIVO... Licenser: scanned 1459 files in 276 directories. Licensed files: 447 Known exceptions: 47 Missing tags: 0 Scanning Vitro... Licenser: scanned 2654 files in 584 directories. Licensed files: 1404 Known exceptions: 543 Missing tags: 0 Licenser was successful engin:releaseScripts jeb228$
When creating the first release candidate, you need to create the maintenance branches for that release.
Test candidates are made from the develop branch. No maintenance branch is created, and a code freeze is not declared.
When creating a "bug fix" release, such as 1.8.1, no branch is created. All fixes to release 1.8 will use the same maintenance branch.
To create the maintenance branches, the script will execute these commands:
cd [repository directory] git checkout develop git pull git checkout -b maint-rel-[release-label]
For example:
cd /usr/local/BuildingReleases/VIVO git checkout develop git pull git checkout -b maint-rel-1.8 cd /usr/local/BuildingReleases/Vitro git checkout develop git pull git checkout -b maint-rel-1.8
This script will not run if the branch already exists, or if you are creating a test candidate.
This script will confirm that you want to create the maintenance branches in VIVO and Vitro, and ask you to approve the sets of commands before they are executed.
engin:releaseScripts jeb228$ ruby 2_create_branches.rb OK to create branches named 'maint-rel-1.8' (y/n) y Creating branches Execute these commands? (in /usr/local/BuildingReleases/VIVO) >>>>> git checkout develop >>>>> git pull >>>>> git checkout -b maint-rel-1.8 (y/n) y >>>>> git checkout develop Already on 'develop' >>>>> git pull Already up-to-date. >>>>> git checkout -b maint-rel-1.8 Switched to a new branch 'maint-rel-1.8' Execute these commands? (in /usr/local/BuildingReleases/Vitro) >>>>> git checkout develop >>>>> git pull >>>>> git checkout -b maint-rel-1.8 (y/n) y >>>>> git checkout develop Already on 'develop' >>>>> git pull Already up-to-date. >>>>> git checkout -b maint-rel-1.8 Switched to a new branch 'maint-rel-1.8' engin:releaseScripts jeb228$
You can confirm that the branches were created by issuing these commands:
cd [repository directory] git branch --list
For example:
cd /usr/local/BuildingReleases/VIVO git branch --list cd /usr/local/BuildingReleases/Vitro git branch --list
In each repository, you should see the name of the new branch in the list.
Every release candidate should be marked with a tag on the maintenance branch. Every test candidate should be marked with a tag on the develop
branch.
To create the tags, the script will execute these commands:
cd [repository directory] git checkout [maintenance branch] git pull git tag -a [tag label] -m '[tag message]'
For example
cd /usr/local/BuildingReleases/VIVO git checkout maint-rel-1.8 git pull git tag -a rel-1.7-rc1 -m 'Release 1.8 rc1 tag' cd /usr/local/BuildingReleases/Vitro git checkout maint-rel-1.8 git pull git tag -a rel-1.7-rc1 -m 'Release 1.8 rc1 tag'
Note that a test candidate is built on the develop
branch, not on a maintenance branch.
Note that the pull
command will not be performed if the branch does not yet exist on the GitHub repository.
This script will not run if you are creating a release candidate or final release, and the maintenance branch doesn't exist. Test candidates do not require a maintenance branch.
This script will make sure that the branch is current, relative to GitHub, and then apply the tag.
engin:releaseScripts jeb228$ ruby 3_create_tags.rb OK to create tags named 'rel-1.8-rc1' 'Release 1.8 rc1 tag' (y/n) y Creating tags Execute these commands? (in /usr/local/BuildingReleases/VIVO) >>>>> git checkout maint-rel-1.8 >>>>> git tag -a rel-1.8-rc1 -m 'Release 1.8 rc1 tag' (y/n) y >>>>> git checkout maint-rel-1.8 Already on 'maint-rel-1.8' >>>>> git tag -a rel-1.8-rc1 -m 'Release 1.8 rc1 tag' Execute these commands? (in /usr/local/BuildingReleases/Vitro) >>>>> git checkout maint-rel-1.8 >>>>> git tag -a rel-1.8-rc1 -m 'Release 1.8 rc1 tag' (y/n) y >>>>> git checkout maint-rel-1.8 Already on 'maint-rel-1.8' >>>>> git tag -a rel-1.8-rc1 -m 'Release 1.8 rc1 tag' engin:releaseScripts jeb228$
You can confirm that the tags were created by issuing these commands:
cd [repository directory] git tag --list [tag name]
For example:
cd /usr/local/BuildingReleases/VIVO git tag --list rel-1.8* cd /usr/local/BuildingReleases/Vitro git tag --list rel-1.8*
In each repository, you should see the name of the new tag.
In order to create the release, a "release directory" is built and then archived for distribution.
The release directory is named after the VIVO release, and the Vitro files are inserted into a vitro-core
subdirectory.
As shown below, the release directory is created as a child of the base directory. It is stored by release number and by release candidate label. The name of the directory also includes the release number and release candidate label, so it will be easily identified after the user unpacks it.
To create the release directory, the script will execute these commands:
rm -Rf [parent of release directory] mkdir -pv [release directory] cp -R [VIVO repository directory]/* [release directory] mkdir -pv [release directory]/vitro-core cp -R [Vitro repository directory]/* [release directory]/vitro-core
For example:
rm -Rf /usr/local/BuildingReleases/release_1.8/rc1 mkdir -pv /usr/local/BuildingReleases/release_1.8/rc1/vivo-rel-1.8-rc1 cp -R /usr/local/BuildingReleases/VIVO/* /usr/local/BuildingReleases/release_1.8/rc1/vivo-rel-1.8-rc1 mkdir -pv /usr/local/BuildingReleases/release_1.8/rc1/vivo-rel-1.8-rc1/vitro-core cp -R /usr/local/BuildingReleases/Vitro/* /usr/local/BuildingReleases/release_1.8/rc1/vivo-rel-1.8-rc1/vitro-core
This script will not run unless the tags have been created.
This script copies the VIVO and Vitro repositories into the release directory. The copy process omits any hidden files at the top level of the repository, so the .git
directories are not copied.
engin:releaseScripts jeb228$ ruby 4_extract_files.rb OK to create export area at /usr/local/BuildingReleases/release_1.8/rc1/vivo-rel-1.8-rc1 ? (y/n) y Building export area Execute these commands? >>>>> rm -Rf /usr/local/BuildingReleases/release_1.8/rc1 >>>>> mkdir -pv /usr/local/BuildingReleases/release_1.8/rc1/vivo-rel-1.8-rc1 >>>>> cp -R /usr/local/BuildingReleases/VIVO/* /usr/local/BuildingReleases/release_1.8/rc1/vivo-rel-1.8-rc1 >>>>> mkdir -pv /usr/local/BuildingReleases/release_1.8/rc1/vivo-rel-1.8-rc1/vitro-core >>>>> cp -R /usr/local/BuildingReleases/Vitro/* /usr/local/BuildingReleases/release_1.8/rc1/vivo-rel-1.8-rc1/vitro-core (y/n) y >>>>> rm -Rf /usr/local/BuildingReleases/release_1.8/rc1 >>>>> mkdir -pv /usr/local/BuildingReleases/release_1.8/rc1/vivo-rel-1.8-rc1 >>>>> cp -R /usr/local/BuildingReleases/VIVO/* /usr/local/BuildingReleases/release_1.8/rc1/vivo-rel-1.8-rc1 >>>>> mkdir -pv /usr/local/BuildingReleases/release_1.8/rc1/vivo-rel-1.8-rc1/vitro-core >>>>> cp -R /usr/local/BuildingReleases/Vitro/* /usr/local/BuildingReleases/release_1.8/rc1/vivo-rel-1.8-rc1/vitro-core engin:releaseScripts jeb228$
You can confirm that the release directory was created by executing this command:
ls [release directory]
For example:
ls -l /usr/local/BuildingReleases/release_1.8/rc1/vivo-rel-1.8-rc1
You should see the usual top-level VIVO entries, plus the vitro-core
directory.
The release process removes the .git
information from the files. This means that the revision information must be hard-coded into the release. This information takes the form of two files names revisionInfo
, one in VIVO and one in Vitro. Each file contains the release name and the abbreviated commit hash that describes the state of the repository when the release was tagged.
The files are stored at these locations:
[vivo]/revisionInfo
[vivo]/vitro-core/revisionInfo
The contents of each file might look something like this:
rel-1.8-rc1 2d48b90
This script will not run unless the tags have been created, and the files have been extracted to the release directory.
This script queries git for the hash of the commits on the tags, and creates the files:
engin:releaseScripts jeb228$ ruby 5_insert_revision_info.rb OK to write revision_info at these paths? /usr/local/BuildingReleases/release_1.8/rc1/vivo-rel-1.8-rc1/revisionInfo /usr/local/BuildingReleases/release_1.8/rc1/vivo-rel-1.8-rc1/vitro-core/revisionInfo ? (y/n) y Building revision info Writing 'rel-1.8-rc1 ~ 2d48b90' to /usr/local/BuildingReleases/release_1.8/rc1/vivo-rel-1.8-rc1/revisionInfo Writing 'rel-1.8-rc1 ~ 69a934e' to /usr/local/BuildingReleases/release_1.8/rc1/vivo-rel-1.8-rc1/vitro-core/revisionInfo engin:releaseScripts jeb228$
The distribution files for a VIVO release are a tar.gz
file and a zip
file of the release directory.
The release of Vitro is also created at the same time. The vitro-core
directory is copied into its own area, named appropriately for the release, and distribution files are created.
To create the distribution files, the script will execute these commands:
cp -r [release directory]/vitro-core [vitro release directory] zip -rq [release directory].zip [release directory] tar -czf [release directory].tar.gz [release directory] zip -rq [vitro release directory].zip [vitro release directory] tar -czf [vitro release directory].tar.gz [vitro release directory]
For example:
cp -r vivo-rel-1.8-rc1/vitro-core vitro-rel-1.8-rc1 zip -rq vivo-rel-1.8-rc1.zip vivo-rel-1.8-rc1 tar -czf vivo-rel-1.8-rc1.tar.gz vivo-rel-1.8-rc1 zip -rq vitro-rel-1.8-rc1.zip vitro-rel-1.8-rc1 tar -czf vitro-rel-1.8-rc1.tar.gz vitro-rel-1.8-rc1
This script will not run unless the revision info file have been created.
This script creates a separate Vitro release directory and then creates archives of both VIVO and Vitro
engin:releaseScripts jeb228$ ruby 6_create_distribution_files.rb OK to create distribution files in /usr/local/BuildingReleases/release_1.8/rc1/vivo-rel-1.8-rc1 ? (y/n) y Creating distribution files Execute these commands? >>>>> cp -r vivo-rel-1.8-rc1/vitro-core vitro-rel-1.8-rc1 >>>>> zip -rq vivo-rel-1.8-rc1.zip vivo-rel-1.8-rc1 >>>>> tar -czf vivo-rel-1.8-rc1.tar.gz vivo-rel-1.8-rc1 >>>>> zip -rq vitro-rel-1.8-rc1.zip vitro-rel-1.8-rc1 >>>>> tar -czf vitro-rel-1.8-rc1.tar.gz vitro-rel-1.8-rc1 (y/n) y >>>>> cp -r vivo-rel-1.8-rc1/vitro-core vitro-rel-1.8-rc1 >>>>> zip -rq vivo-rel-1.8-rc1.zip vivo-rel-1.8-rc1 >>>>> tar -czf vivo-rel-1.8-rc1.tar.gz vivo-rel-1.8-rc1 >>>>> zip -rq vitro-rel-1.8-rc1.zip vitro-rel-1.8-rc1 >>>>> tar -czf vitro-rel-1.8-rc1.tar.gz vitro-rel-1.8-rc1 engin:releaseScripts jeb228$
When the final release is created, the maintenance branch is merged to the master
branch, and the release is tagged. This is the only time that the master
branch is modified, so a commit to the master
branch always implies a final release.
The maintenance branch is not deleted by this operation, and remains available for creating bug-fix releases.
In order to merge the branch and create the tag, the script will execute these commands:
cd [repository directory] git branch master origin/master git checkout master git merge --no-ff -Xtheirs [branch] git tag -a -f [tag] -m '[message]'
For example:
cd /usr/local/BuildingReleases/VIVO git branch master origin/master git checkout master git merge --no-ff -Xtheirs maint-rel-1.8 git tag -a -f rel-1.8 -m 'Release 1.8 final tag' cd /usr/local/BuildingReleases/Vitro git branch master origin/master git checkout master git merge --no-ff -Xtheirs maint-rel-1.8 git tag -a -f rel-1.8 -m 'Release 1.8 final tag'
Note that the branch
command will only be executed if the cloned repositories are not already tracking the master
branch on GitHub.
This script will not run unless this is a final release, and the branches have been created
The script will merge the maintenance branch to the master
branch, and move the tag to the master
branch. Here is an excerpt of a sample run:
engin:releaseScripts jeb228$ ruby 7_merge_to_master.rb OK to merge the rel-1.8 tags to the master branches? (y/n) y Merging tags Execute these commands? (in /usr/local/BuildingReleases/VIVO) >>>>> git branch master origin/master >>>>> git checkout master >>>>> git merge --no-ff -Xtheirs maint-rel-1.8 >>>>> git tag -a -f rel-1.8 -m 'Release 1.8 final tag' (y/n) y >>>>> git branch master origin/master Branch master set up to track remote branch master from origin. >>>>> git checkout master Checking out files: 100% (691/691), done. Switched to branch 'master' >>>>> git merge --no-ff -Xtheirs maint-rel-1.8 Removing utilities/load-testing/uploadFileFaker/property_file_reader.rb Removing utilities/load-testing/uploadFileFaker/UploadFileFaker.rb Removing utilities/load-testing/tests/personUris.csv Removing utilities/load-testing/tests/infoResourceUris.csv Auto-merging themes/wilma/templates/identity.ftl Auto-merging themes/wilma/css/wilma.css ... many more lines like this ... Merge made by the 'recursive' strategy. .gitignore | 10 + README.md | 58 + build.xml | 23 +- config/licenser/known_exceptions.txt | 1 + config/licenser/licenser.properties | 3 - ... many more lines like this ... 655 files changed, 26005 insertions(+), 61688 deletions(-) create mode 100644 .gitignore create mode 100644 README.md create mode 100755 doc/licenses/leaflet.LICENSE.txt delete mode 100644 doc/list_view_configuration_guidelines.txt create mode 100644 doc/other_servlet_containers.html ... many more lines like this ... >>>>> git tag -a -f rel-1.8 -m 'Release 1.8 final tag' Updated tag 'rel-1.8' (was 1c923dc) Execute these commands? (in /usr/local/BuildingReleases/Vitro) >>>>> git branch master origin/master >>>>> git checkout master >>>>> git merge --no-ff -Xtheirs maint-rel-1.8 >>>>> git tag -a -f rel-1.8 -m 'Release 1.8 final tag' (y/n) y >>>>> git branch master origin/master Branch master set up to track remote branch master from origin. >>>>> git checkout master Checking out files: 100% (1159/1159), done. Switched to branch 'master' >>>>> git merge --no-ff -Xtheirs maint-rel-1.8 warning: CRLF will be replaced by LF in webapp/config/example.build.properties. The file will have its original line endings in your working directory. warning: CRLF will be replaced by LF in webapp/config/example.runtime.properties. The file will have its original line endings in your working directory. Auto-merging webapp/web/templates/freemarker/edit/forms/rdfsLabelForm.ftl Auto-merging webapp/web/templates/freemarker/edit/forms/pageManagement.ftl Auto-merging webapp/web/templates/freemarker/edit/forms/pageManagement--sparqlQuery.ftl Auto-merging webapp/web/templates/freemarker/edit/forms/pageManagement--fixedHtml.ftl Auto-merging webapp/web/templates/freemarker/edit/forms/pageManagement--classIntersections.ftl ... many more lines like this ... Removing utilities/testrunner/selenium/selenium-ide-1.0.12.xpi CONFLICT (modify/delete): webapp/src/edu/cornell/mannlib/vitro/webapp/filters/WebappDaoFactorySDBPrep.java deleted in maint-rel-1.8 and modified in HEAD. Version HEAD of webapp/src/edu/cornell/mannlib/vitro/webapp/filters/WebappDaoFactorySDBPrep.java left in tree. Removing utilities/testrunner/selenium/ide-extensions.js Removing utilities/buildutils/src/edu/cornell/mannlib/vitro/utilities/revisioninfo/InfoResponseParser.java Auto-merging solr/homeDirectoryTemplate/conf/schema.xml Removing solr/apache-solr-3.1.0.war Automatic merge failed; fix conflicts and then commit the result. Command failed: code 1 - Aborting. engin:releaseScripts jeb228$
It's not clear why this conflict is created. Surely the master branch has not been modified since the previous release? Perhaps it is a result of the shenanigans that were used when converting from Subversion to Git.
I got past this point by reading the CONFLICT
messages, resolving the conflict, committing the merge, and manually issuing the tag command.
For each release product – not just the final release – the branches and tags should be pushed back to GitHub, so they will be available to those sites that prefer to work directly from the repositories.
To push the changes to GitHub, the script will execute these commands:
cd [repository directory] git push --all git push --tags
For example:
cd /usr/local/BuildingReleases/VIVO git push --all git push --tags cd /usr/local/BuildingReleases/Vitro git push --all git push --tags
The script will push any branches, tags or merges back to the GitHub repositories.
jeb228$ ruby 8_push_changes.rb OK to push changes to the origin? (y/n) y Merging tags Execute these commands? (in /usr/local/BuildingReleases/VIVO) >>>>> git push --all >>>>> git push --tags (y/n) y >>>>> git push --all Everything up-to-date >>>>> git push --tags Counting objects: 1, done. Writing objects: 100% (1/1), 167 bytes, done. Total 1 (delta 0), reused 0 (delta 0) To git@github.com:j2blake/VIVO.git * [new tag] rel-1.8-tc2 -> rel-1.8-tc2 Execute these commands? (in /usr/local/BuildingReleases/Vitro) >>>>> git push --all >>>>> git push --tags (y/n) y >>>>> git push --all Everything up-to-date >>>>> git push --tags Counting objects: 1, done. Writing objects: 100% (1/1), 168 bytes, done. Total 1 (delta 0), reused 0 (delta 0) To https://github.com/j2blake/Vitro.git * [new tag] rel-1.8-tc2 -> rel-1.8-tc2 jeb228$
After the files have been created and the changes have been pushed to GitHub, the release product must be publicised.
For now, we are storing release candidate files in
rollins.mannlib.cornell.edu:/var/www/html/download/
The final release files must be posted on SourceForge.
Send an announcement e-mail to vivo-release-test@lists.sourceforge.net
For a final release, more must be done. Check the Release process page.