Page History

TimMark DonohueWood [210:00 PM]
@here: ItWelcome to the weekly developer meeting, all!
There's oura link weeklyto DSpacethe DevMtgagenda timeabove.

Terry Agenda is at https://wiki.duraspace.org/display/DSPACE/DevMtg+2019-02-27
As usual, let's do a quick roll call to Brady [10:01]
Hello

Mark Wood [10:01]
Shall we see who else is able to join the meeting today

Mark Wood [2:00 PMhere?
It seems that we are three today.

Alexander Sulfrian [10:02]
Hi

TimTerry DonohueBrady [210:01 PM03]
HiThe @mwood.Docker for Well,Repository itManagers might bewebinar awent very quickwell meetingyesterday. if it's just the two of us.  :slightly_smiling_face:
I'll go ahead an post some general updates here (just for anyone listening or looking in on notes later).  After that, we can see if anyone else has popped in
First off, a general status update on DSpace 7 (for anyone interested).  We are aiming for a 7.0 Preview Release (not all features, but many main ones) in late March.  The Beta will follow that sometime in May (and Beta will be the first release including all DSpace features)

James Creel [2:04 PM]
Hey gang https://duraspace.org/webinar/ (edited) 

Mark Wood [10:03]
Quick reminders:  the DSpace 7 and Entities working groups are still working, and do continue with scheduled meetings, although some recent meetings have been cancelled due to conflicts.
Hi @sulfrian, we are four now.
Anything else to say about the webinar, @terrywbrady?

Terry Brady [10:05]
We had 55 people and a dozen great questions at the end.
One attendee mentioned that they were able to work through the sample exercises while we were on the call.  I hope to hear more from other attendees over the next week.

Mark Wood [210:04 PM07]
Hey, welcome.

Tim Donohue [2:05 PM]
These 7.0 timelines have slipped slightly cause of our volunteer workforce (we are all volunteers or on donated time, and sometimes that means features take a bit longer to create or review).  However, we're working hard to ensure Beta is ready for user testing / training at OR2019.
Expect 7.0 Final sometime in July/Aug timeframe. The final release date will be easier to nail down after we run a community testathon (to see what bugs/issues still exist) on the Beta.
I think that's the 7.0 updates at a high level.  Obviously though, for the nitty gritty details, check the DSpace 7 Working Group meeting notes or join a meeting.

Kim Shepherd [2:07 PM]
hi all, i'm semi-here

Tim Donohue [2:07 PM]
Hi James & Kim, welcome
Any questions on 7.0 updates?

Mark Wood [2:09 PM]
Bringing a beta to OR could be advantageous:  a room full of testers with developers right there handy....

Kim Shepherd [2:09 PM]
sounds good to me

Tim Donohue [2:09 PM]
Oh, I should also mention there's a new Marketing Working Group just getting started.  I know this is a DevMtg, but if you know folks interested in marketing/promotion, pass this along: https://wiki.duraspace.org/display/DSPACE/DSpace+7+Marketing+Working+Group
Ok, moving right along then.
Regarding 6.x updates (i.e. an eventual 6.4), the status hasn't changed much recently.  I'll admit, all my effort is currently going towards helping get 7.0 out the door as soon as we can.  I'm sure 6.4 will happen, but it's mostly waiting on someone (a Committer) to pick it up and run with it.
If anyone has more to add on a 6.4, or has any comments/questions, feel free to share them. That's the extent of the update though

James Creel [2:12 PM]
Not sure if this is exactly on topic, but I've got a librarian with a concern about support for deleted objects with OAI and it led me here: https://wiki.duraspace.org/pages/viewpage.action?pageId=68064778
and to this closed card: https://jira.duraspace.org/browse/DS-2491
Anyway, they would like "persistent" support for the deleted items which would mean some permanent tombstones for deleted stuff.  Sounds like a fairly big architectural commitment.   If we got a card for this on a sprint, might it go in to 6.x or 7's OAI?
If this seems like a bad idea I could communicate that

Kim Shepherd [2:14 PM]
a change in how OAI works wouldnt make it into a 6.x as it's not a bugfix
but could conceivably be part of 7.0 if we could figure out a good way to support it.. at the moment, expunged items are hard to keep track of on account of they really are completely gone :wink:

Tim Donohue [2:15 PM]
@jcreel256: that sounds like a significant change.  Obviously, the reason OAI works the way it does is that the underlying object is gone when something is deleted.  The only way to generate a "tombstone" is a withdrawal at this time.

Kim Shepherd [2:16 PM]
i've got some OAI improvements/fixes (both PMH and harvest) to contribute sometime, but it'll be a mix of fixes for 6.x and suggestions for 7.x (i've been working with harvesting a fair bit lately)
i'd suggest the librarian use withdrawing as the default 'deletion' method for now?

James Creel [2:16 PM]
Yes, makes perfect sense.

Tim Donohue [2:17 PM]
To be completely honest, the "window of opportunity" for 7.0 changes is closing rapidly.  We are forced to really tighten our scope to ensure we hit that Beta goal for OR2019.  So, anytime I hear of suggestions that sound major, I will be poking at them / pushing back on them.
Minor changes are still obviously welcome though

James Creel [2:17 PM]
Also sensible.

Mark Wood [2:18 PM]
Yes, I doubt there would be time available to handle a new 7.0 feature unless it is really simple, easy to review, and already written.

James Creel [2:19 PM]
On the bright side, the understanding that new features don't really have much of an opening right now could motivate those who want new stuff to help get the 7.0 out the door faster.
I can express this to my stakeholders around here.

Tim Donohue [2:20 PM]
Yes, exactly.  :slightly_smiling_face:   To be clear, 7.0 already has a ton of amazing new features.  But, obviously, no release can do everything, and we're already actively limiting any additional "new features" from going into 7.0 (unless they are very small / self contained)

Mark Wood [2:21 PM]
And I've already started posting ideas for things to do in 8.0.

Tim Donohue [2:22 PM]
Thanks for that @mwood.  I have been following your thinking/emails on dspace-devel.  I just haven't found time to comment...but the ideas/brainstoms are all good

Mark Wood [2:22 PM]
Anyway, it sounds like this is something that DSpace could/should address.

Tim Donohue [2:22 PM]
agreed
So, this sounds like a wrapped up topic.  Any other final notes/questions on 6.x?
Moving along then.
Topic #3: So, I noticed that (yet again) I have a meeting conflict next week.  February's 28 days means this conflict came up both in Feb and now in March...but it won't happen again for the foreseeable future (I checked)
So, this means, I won't be able to attend this meeting next week (March 3 at 15UTC).  Anyone want to take chair responsibility next week?  Or should we just cancel and touch base again on March 13? (edited) 
Sorry, March 6
My days are totally wrong.  I'll miss March 6 at 15UTC

Mark Wood [2:25 PM]
I can take the chair, if we think there is anything to discuss next week.

Tim Donohue [2:27 PM]
To be honest, the topics in this meeting have mostly been updates as of late.  I'm not sure if that means this meeting is becoming *less useful* (in light of many other DSpace meetings in a week, mostly around DSpace 7), or if folks find it *more useful* (as it's one meeting to attend for a summary of what's going on).

Mark Wood [2:28 PM]
There are things (like the tombstone issue) that don't really fit into the more specialized meetings.  And there are a number of WGs that will dissolve over the next few months so, fewer meetings....

Tim Donohue [2:28 PM]
But, beyond those usual "updates", I don't have specific topics for next week (just like I don't really have other specific topics this week).  I do welcome anyone here to bring topics to the agenda though.

James Creel [2:29 PM]
Depends on your perspective, I'm sure.  This meeting is sort of my speed since the 7 meetings would be on specific cards.

Kim Shepherd [2:29 PM]
if someone does pick up coordination of 6.4, this meeting is a pretty good place for 6.x release updates

Tim Donohue [2:31 PM]
Yes, to be clear, I'm not trying/implying canceling this meeting altogether.  I'm just pointing out that this meeting is sorta a "weekly update meeting" of sorts right now.  So, when it comes to specific topics for next week, I don't have any (other than I can pass along general updates to @mwood to report).
And I'm also wanting to note this "weekly update meeting" status out loud, as I've been unable to find/bring more specific topics to this meeting.  But, I welcome others doing so in future weeks :slightly_smiling_face:

Mark Wood [2:33 PM]
It may be time to start thinking seriously about "what do we do after 7 releases?" and this would be a place to do it, if rapid interaction is needed.

Tim Donohue [2:33 PM]
And as @kshepherd notes, this definitely is the perfect meeting for 6.x questions/discussions.  I've also tried to bring "backend" updates, like Solr upgrade & "One Webapp" refactor here...simply cause those are easy to also understand from a 6.x perspective.
@mwood: yes, that's true too. I haven't had much headspace for that quite yet :wink:  But, I agree that the time is approaching
In any case, this was all a bit of a tangent.... The main question is whether to meet next week.  @mwood if you want to lead, I can do my best to pass along topics (though ping me if I forget).  If there's no specific topics/updates, you all could also just have a "bring your own topic" style meeting

Mark Wood [2:36 PM]
Do we want a meeting next week, then?  If so, I can lead.

Tim Donohue [2:37 PM]
Since this is a "weekly update meeting" and I won't be there, I think it's a question of who will attend & what would you like to talk about.  Or, just take the week off, if you want the hour to work on DSpace stuff :wink:  Honestly, either way is fine by me

James Creel [2:37 PM]
I can report back with my librarians' comments on the "transient" status of OAI responses for deleted items versus withdrawn ones
For what it's worth, the use case is our being harvested by EBSCO for EDS (EBSCO Discovery Service) and how they have some items in there that we deleted.  The current ticket with them is still up in the air.
And who knows if they would even care if we had better compliance with OAI in this regard.

Tim Donohue [2:39 PM]
That seems like a fine topic to touch back on in more detail, if there's interest.  Perhaps the goal could be to turn that use case into a JIRA ticket (that we can then work to schedule out)

Kim Shepherd [2:39 PM]
i'll be asleep at 15UTC so i'm neutral :wink:

Tim Donohue [2:40 PM]
Typical @kshepherd, always needing to sleep at 3am :laughing:
(Definitely don't get up and work at 3am.  I never would)

Kim Shepherd [2:41 PM]
heh

Tim Donohue [2:41 PM]
Sounds like the meeting is "on", @mwood will lead it.  You have a topic.  Feel free to wrap it up early though if you run out of things to discuss.  (And I'll catch up later in the logs)

Kim Shepherd [2:41 PM]
quick note from me regarding solr and docker change testing - i still haven't finished doing this properly sorry, but i'll update PRs when i can

Tim Donohue [2:42 PM]
Moving along now...yes, we can move into Solr upgrade updates from @mwood: https://github.com/DSpace/DSpace/pull/2058
Anything to update us on this week?

Mark Wood [2:42 PM]
No, I've been waiting to hear what folks think of it.

Tim Donohue [2:43 PM]
Ok, I'll admit, I haven't had time to look back at it myself either.  I know Terry has been out a bit the last week or so.  And Kim just said he hasn't had a chance.

Mark Wood [2:43 PM]
Understood.
I should go ahead and start tinkering with how to address existing sites, especially sharding.

Tim Donohue [2:44 PM]
So, it sounds like we're waiting on reviewers.  I do also know that (in the DSpace 7 meeting last week ) Art also volunteered to look at it. So, maybe we'll get an update on that tomorrow.
So, not hearing anything to discuss further here.  I guess we'll move along
On the "One Webapp" backend side of things..https://github.com/DSpace/DSpace/pull/2265  I'll note that as of *today*, I have OAI-PMH also working in the merged webapp.  That means both SWORDs (v1 and v2) and OAI-PMH are all running alongside RESTv7 in Spring Boot.
The final piece of the puzzle is RDF...and a little bit of cleanup/tidying of the code
OAI-PMH proved to be the hardest piece so far, as OAI-PMH has its own (basic) integration tests. I had to move those over & convert them into Spring Boot compatibility.  But, they all now work.  And both SWORDs have basic ITs that also work
That's the update. The code is all there to have early reviews. You can even try it out if you wish.  Feedback is more than welcome.  I'm hoping RDF will prove to be a bit easier, and maybe this will be ready for official reviews in the next week.
Any questions or comments?

Mark Wood [2:50 PM]
Am I correct in thinking that, if one *did* still want to separate services (onto different boxes for instance) one could just run multiple copies of the Single Webapp with different bits enabled?

Tim Donohue [2:52 PM]
@mwood: kinda.  However, currently you cannot turn "off" REST API v7.  Everything else can be turned on/off though.  So, if you deploy this webapp multiple times, you'd have multiple `/api` endpoints

Mark Wood [2:52 PM]
Ah.  Thanks.

Tim Donohue [2:53 PM]
We might be able to look into treating REST API v7 similarly (and have the `/api` endpoint turn on/off), but I wasn't planning to do that as part of this initial PR.

Mark Wood [2:53 PM]
Sensible.

Kim Shepherd [2:55 PM]
Before we end, i just wanted to note that I might start a JIRA ticket or wiki page or something to discuss potential cleanup of xoai code and dspace-oai... i've run into a few things which i think could be refactored, eg. more standard/featureful xml I/O, better hibernate usage, some improvements to the base xoai xml format and so on
so that could maybe be a place to discuss ways to support other OAI-PMH functionality, etc.

Mark Wood [2:55 PM]
One thing we ought to do at some point is move to XOAI v4.

Tim Donohue [2:56 PM]
@kshepherd: yes, that'd be welcome.  We might want to start with a wiki page.  I also noticed there's a bit of "crud" in the OAI-PMH codebase as I worked to pull it into Spring Boot (in that "one webapp" PR)
And yes, we really should look at moving to XOAI v4 (latest version) instead of still using XOAI v3.
(I think there's a ticket about moving to XOAI v4 already)

Kim Shepherd [2:57 PM]
@mwood yeh, that'd be part of it -- quite a lot of the stuff i want to fix is in xoai codebase rather than dspace-oai
yep https://jira.duraspace.org/browse/DS-2595 is the "let's do xoai v4"

Tim Donohue [2:58 PM]
We are getting near the top of the hour, so I won't call any more topics (plus, since Terry is out today, it's hard to do Docker updates)
@kshepherd: yes, I'd like to get us up to XOAI v4 first (ideally) and then fix that version of the codebase.  The way v3 works is a bit "odd" to me in some parts, and at least (at a glance) v4 looks cleaner (but I don't have experience to say whether that's actually true)
@kshepherd: in any case, a wiki page to *start* the discussion (and start gathering info/details/ideas) seems like a great place to begin.

Mark Wood [3:00 PM]
Quick question:  DS-3989 is at +2 and has been for some time.  There was talk about beefing up the tests, but it didn't block approval.  Doing ITs for this kind of code is extremely difficult, and I'm inclined to merge this now and improve testing later.  Thoughts?

Tim Donohue [3:01 PM]
Sorry, had to find the actual PR.  This looks to be it: https://github.com/DSpace/DSpace/pull/2180

Kim Shepherd [3:01 PM]
sounds reasonable to me..

Tim Donohue [3:02 PM]
I'm OK with that direction, I see that @kshepherd had been assigned as a reviewer on this too...but Terry & Ben already gave it +1's.  So, if everyone else is OK with it, I don't see a reason to hold it up.

Mark Wood [3:03 PM]
Thanks.

Tim Donohue [3:03 PM]
Although, it probably does need some documentation if it changes any user behavior.  If it's all "underneath" refactoring, then that may not be necessary

Mark Wood [3:03 PM]
I will see to documenting it.
The Jira won't be closed until that's done.

Tim Donohue [3:04 PM]
Thanks!  yes, that makes sense to move the Documentation part to JIRA and just note there that the ticket is waiting on final Docs to be created
OK, well, we are over time now (by ~5 mins).  So, let's close up today's meeting.

Kim Shepherd [3:05 PM]
cheers all

Tim Donohue [3:05 PM]
I'll see you all (in this meeting) in two weeks! But, next week, @mwood will lead the meeting, you'll get an update from @jcreel256 on OAI-PMH persistence, and anyone else can bring topics too!
thanks all!

Mark Wood [3:06 PM]
'bye, all.

James Creel [3:06 PM]
Adios!That does sound as though it is working well.
Since we're talking Docker anyway, shall we skip ahead and see if there is anything else to discuss on that topic today?

Terry Brady [10:09]
Sounds good.  I merged a PR yesterday on master that allows update sequences to run via the database command.  I plan to port that to 5x and 6x.  @tdonohue cautioned that it might not port as easily to 4x.

Mark Wood [10:10]
I see that several of the PR links I copied are to merged PRs.  I'll update the lists after the meeting.

Terry Brady [10:11]
The Docker image we have for Oracle is handy.  I have reached out to Atmire to see if they would want to chat about support for the dspace-oracle image.  It would be great to make that a pluggable option into the existing docker compose files.
@tdonohue shared a link to the entities wg dataset.  I plan to explore how that could be used as a sample AIP loaded dataset for Docker images.
That is it for me on Docker.

Mark Wood [10:12]
Comments from others?
OK, we'll move on.  Back to the top of the list:  I have no status updates for 6_x or 7_x.

Tim Donohue [10:14]
Sidenote for @terrywbrady (sorry, in another meeting, so just lurking):  We may not be able to create AIPs of the Entities dataset -- as all aspects of Entities (especially relationships between them) are not yet represented in AIPs.  It's worth a try, but not sure if it'll work

Mark Wood [10:15]
Comments from others on those?
None, it seems.  On the Solr upgrade:  https://github.com/DSpace/DSpace/pull/2058 has had another review, and has an open change request.  What does it need to make it approvable?
I will spend some time today on seeing what is needed for upgrading an older DSpace instance and moving its cores to a separate Solr install.

Terry Brady [10:19]
Sorry that I did not update my review.  I think I was waiting to see if you needed another test from me.

Mark Wood [10:20]
I mainly need to have addressed your concerns so that the request can be closed.
My thanks to everyone who has reviewed it.

Terry Brady [10:21]
You had addressed all but the statistics issue before my prior review.  I have not yet re-validated your fix to the statistics object.
I'll do another quick look at the code.

Mark Wood [10:22]
OK, thank you!
Other comments on the Solr upgrade?
OK, moving on:  DSpace Backend as one webapp.  No updates that I know of, and Tim is unavailable at the moment.  Reviews would be helpful, I am sure.  (I should review it.)

Tim Donohue [10:25]
A quick update, the PR is fully ready for review: https://github.com/DSpace/DSpace/pull/2265

Mark Wood [10:25]
Ah, thank you!

Tim Donohue [10:25]
see latest comment there for more details

James Creel [10:26]
I've got to run to another meeting in 5 minutes - I might jump in and give an update on support for deleted objects in OAI.

Mark Wood [10:26]
OK, please do.

James Creel [10:27]
So the librarian stakeholders here are keen to get "persistent" support for deletions in OAI.  Currently DSpace is supposed to do this for withdrawn items but not for deleted ones which have not tombstone.
Now, I tested the withdrawal use case on a DSpace 6.3 build and it wasn't working for me, so there may be a bug.  I'll investigate further to confirm.
Supporting it for actually deleted items would involve storing some persistent tombstone.  Would others find this objectionable in principle?
Probably would involve change to db schema

Mark Wood [10:29]
How does one actually remove an item so that it is actually gone, no longer in evidence?  Someone is going to want that.

James Creel [10:29]
Right - I agree.  But it fundamentally conflicts with achieving "persistent" support for deletions in OAI.

Mark Wood [10:29]
A "tombstone record" could be an Item with no content streams.

James Creel [10:30]
This drives people crazy with Fedora, how you have to delete the tombstones
Perhaps it could be configurable
Anyway, I have got to run.  I'll jump back on this channel or dev channel in just a few minutes.  Want to mention that we will be doing a DSpace sprint here at TAMU starting March 25.

Mark Wood [10:31]
We probably need some new terms.  To me, something deleted is no longer in existence.  We need a term for "no longer offered but you can still know that we had it."

Alexander Sulfrian [10:32]
This does work for withdrawn items.

Mark Wood [10:32]
I would have said that "withdrawn" denotes that state.
I should look and see whether a withdrawn Item can be recovered, i.e. un-withdrawn.  That would be yet another state.

Alexander Sulfrian [10:34]
This is a withdrawn item in OAI: https://refubium.fu-berlin.de/oai/request?verb=GetRecord&metadataPrefix=oai_dc&identifier=oai:refubium.fu-berlin.de:fub188/22673

Terry Brady [10:34]
Pasted image at 2019-03-06, 7:34 AM 


Mark Wood [10:36]
So possibly desired states of an item:  submitted, not yet available; in-archive; withdrawn (we have it but you can't get it); deleted (we had it but it is gone); purged (no longer detectable).
Not to mention embargoed, private....

Terry Brady [10:37]
Do we consider the message that displays for a withdrawn item to be a tombstone?
Pasted image at 2019-03-06, 7:37 AM 


Mark Wood [10:37]
For completeness:  submission in development.
Perhaps.  I'm still unclear on whether a tombstone belongs to something withheld from view, something actually no longer in the archive, or both.
The use of the word makes me think it means dead-and-gone, no longer in the archive.

Terry Brady [10:40]
These states would be nice to document clearly for end-users.  If I remember correctly, there are 3 booleans in the item table and a couple of them seemed redundant to me until I had used DSpace for several years.

Mark Wood [10:41]
That probably means that we should represent them as states, as in positions in a state diagram, rather than separate booleans.
But we also need a clear understanding of what the community wants, so that we are implementing *useful* states.

Terry Brady [10:42]
The most recent discovery to me was the representation of item templates in the item table.

Mark Wood [10:42]
That probably *is* a case for a boolean.
Hm, what we have is in_archive, withdrawn, discoverable.

Alexander Sulfrian [10:45]
First step could be to create a state diagram of the current possibilities and then discuss changes afterwards.

Mark Wood [10:45]
That seems good.
We then need more information from librarians on what they want to do.

Terry Brady [10:47]
I imagine that any changes to this behavior should be introduced in a major release rather than in a point release... unless there is no clear sense of expected behavior.

Mark Wood [10:47]
Yes.
But, good documentation of what DSpace does now could be added anytime.

Terry Brady [10:49]
Definitely.  Where (within the wiki) is the best location for this documentation?

Mark Wood [10:49]
Good question....

Terry Brady [10:49]
It would be nice to document the anticipate behavior of all 8 (2^3) states.

Mark Wood [10:49]
Using DSpace | Items and Metadata?

Terry Brady [10:50]
That sounds like a smart place.
I need to jump away for 5 min.  I'll loop back if you all have any questions for m.e

Mark Wood [10:50]
OK.
I don't want to interrupt discussion, but we have about 9 minutes left, so if there are other topics we should bring up, this is about the time to request the floor.
Otherwise we can continue on tombstones, or wrap up early.
I will mention that the Entities work brings new possibilities, because Item will now have a "type".  So, for example, "template Item" could be a type.
Have we any further discussion today?

Terry Brady [10:55]
I'm back.

Mark Wood [10:55]
Does anyone want to take on drawing the Item state diagram?

Alexander Sulfrian [10:55]
I can try to draft something.

Mark Wood [10:56]
Thank you.
Should we close the meeting, or are there any final topics?

Tim Donohue [10:57]
Quick sidenote. If you haven't seen it recently, there's a list of DSpace item states here: https://wiki.duraspace.org/display/DSDOC6x/DSpace+Item+State+Definitions

Mark Wood [10:57]
Aha!

Tim Donohue [10:57]
So, any diagram should go there. Updates welcome too

Terry Brady [10:58]
A table showing the combinations of the 3 flags would be good as well.

Mark Wood [10:58]
We should make certain that there is a link to that from somewhere in Using DSpace.
Our time is ended.  Unless there are objections, I'll close the meeting now.

Terry Brady [11:00]
Have a good week!

Mark Wood [11:00]
Thank you all.  Meeting adjourned.