2017-08-10 DSpace 7 Working Group Meeting notes

@here: Welcome, it's time for our weekly check-in on DSpace 7 efforts. As always we'll kick things off with Angular UI updates (here's our current board status: https://waffle.io/DSpace/dspace-angular)

[10:01]
@art-lowel do you want to take it away?

Art Lowel [10:01 AM]
Sure

[10:01]
Since our last meeting I’ve merged both @atarix83's PRs:

[10:01]
one to fix the issues with pagination:

[10:01]
https://github.com/DSpace/dspace-angular/pull/132

[10:01]
and one to rename the retrieve links for bitstreams:

[10:01]
https://github.com/DSpace/dspace-angular/pull/131

[10:02]
@jonas-atmire has been learning angular development

[10:02]
I added him to our github team, and asked him to work on the Mock SearchService ticket

Tim Donohue [10:02 AM]
welcome @jonas-atmire! I saw your first PR came in today :wink:

Art Lowel [10:02 AM]
Here’s the PR:

[10:02]
https://github.com/DSpace/dspace-angular/pull/137

Jonas Van Goolen [10:02 AM]
joined #angular-ui by invitation from @tdonohue

Art Lowel [10:03 AM]
It has a search method that will use the first 10 items of the PubMed Europe collection on the REST demo as mock results, and return them in a random order, with random hit highlights

[10:03]
I’ve reviewed it, and it looks good, but as always more reviewers are welcome

[10:04]
Once this is merged, we can use it to start work on the Simple Search ticket:

[10:04]
https://github.com/DSpace/dspace-angular/issues/136

Tim Donohue [10:04 AM]
Just a simple question on PR 137, is there a reason why the mocks are in code in this PR, instead of in https://github.com/DSpace/dspace-angular/tree/master/src/backend/data/ (alongside other mock data) (edited)

Art Lowel [10:05 AM]
yes, I asked for that in the ticket: https://github.com/DSpace/dspace-angular/issues/134

[10:05]
because the exact spec for the REST responses hasn’t been agreed upon yet

Tim Donohue [10:06 AM]
Oh, I see. Is there a next step to move mocks into /src/backend/data/ (as that endpoint's specs are agreed on)?

Art Lowel [10:06 AM]
and because it would mean our rest backend would have to become a lot more complicated

[10:06]
(mock some endpoints, proxy others)

[10:06]
>Is there a next step to move mocks into /src/backend/data/

[10:06]
yes

[10:06]
when we start work on the rest services for search

[10:07]
https://github.com/DSpace/dspace-angular/issues/135

Tim Donohue [10:08 AM]
Ok, I just wanted to be sure that work was tracked, and that we were continuing the process of creating mocks in /src/backend/data/ (as I think that does a good job of documenting our expectations from the REST endpoint, and is useful for Continuous Integration tests)

Art Lowel [10:08 AM]
I agree

Tim Donohue [10:08 AM]
That all said, PR 137 looks good :+1:

Art Lowel [10:08 AM]
I’m still working on the browse component, no PR yet

[10:09]
but while working on it, I noticed that the function `isUndefined` from node’s util package was used in a few places

[10:09]
Node has deprecated al those `isSomething` methods in that package

[10:10]
and we have our own empty.util.ts file, containing a number of similar functions, so I added is(Not)Undefined and is(Not)Null to that file:

[10:10]
https://github.com/DSpace/dspace-angular/pull/138

[10:10]
After this gets merged, please use those versions instead of the node defaults for future code

[10:11]
The issue with @ngrx/platform that was blocking our upgrade has been fixed

[10:11]
https://github.com/ngrx/platform/issues/97

[10:11]
But the next version hasn’t been released yet, when that happens we can continue with the @ngrx upgrade (edited)

[10:11]
I’ll keep an eye on it

Tim Donohue [10:11 AM]
sounds good

Art Lowel [10:12 AM]
I’ve also come across another ngrx issue, in which they plan to include the rehydration step (meaning: using the server’s store on the client) in the library (edited)

[10:12]
https://github.com/ngrx/platform/issues/101

[10:12]
I’ll be keeping an eye on that issue as well

[10:12]
Those were my updates for today

Tim Donohue [10:13 AM]
Sounds great. I'd say the PRs 137 and 138 look good as soon as you are satisfied (and they pass builds/tests, which it looks like they do)

[10:14]
One other thing I wanted to bring up (mentioned previously in this channel) is that the Universal team seems to be moving forward on creating a Java backend (as an alternative to Node.js):
https://github.com/angular/universal/issues/280#issuecomment-317887154

[10:15]
Just wanted to be sure that others are tracking this work...as it progresses, I'd like us to do more testing to see whether it'll better meet our needs (considering it'd let us use tools we are already comfortable with, being Maven, JVM, etc)

Art Lowel [10:16 AM]
I saw your mention of it earlier, and I’m subscribed to the issue as well

Tim Donohue [10:16 AM]
excellent. Just wanted to bring it up again here in case anyone overlooked it :wink:

[10:17]
Are there any other Angular UI related updates anyone else wants to share? Or any questions on progress or how to get involved?

Andrea Bollini [10:17 AM]
@atarix83 is on holidays for two weeks he is working on https://github.com/DSpace/dspace-angular/issues/133 (edited)

[10:18]
this will be resumed when he comes back

Tim Donohue [10:18 AM]
Oh, and welcome back @bollini! (at least I assume you are back from holiday?)

[10:18]
And thanks for the update about @atarix83's upcoming schedule

Andrea Bollini [10:19 AM]
mmm, not really :slightly_smiling_face: I'm still in holiday just a slow restart as I will be back next week

Tim Donohue [10:19 AM]
Oh, ok. Well, we will welcome you back again next week then :wink:

Andrea Bollini [10:20 AM]
I have also some internal notes to work on for the submission, I will work on that to create the initial tickets for both the angular than the rest part

Tim Donohue [10:21 AM]
Another update from my end. The next Angular Training opportunity is coming up at the Georgetown DSpace User Group meeting (Aug 22-23). I believe the registration closes *tomorrow* (@terrywbrady can correct me if I'm wrong)

Tim Donohue [10:21 AM]
I do want to mention that I'll be updating the training materials from OR2017 next week to prep for that. I'd also appreciate ensuring both our demo sites (Angular & REST) are up to date as of Aug 22, if possible
2 replies Last reply 7 days ago View thread

Andrea Bollini [10:22 AM]
my team have worked on summarize the needs in terms of REST endpoints for the backend functionalities that are presented here: https://wiki.duraspace.org/display/DSPACE/Backend+functionalities

[10:22]
I will be at work, so if you have any issue with the rest demo just let me know

Tim Donohue [10:23 AM]
Ok, one last call for final updates/questions on Angular UI efforts. Anyone? (If not, we can move over to #rest-api for REST updates shortly)

Andrea Bollini [10:24 AM]
have you had a chance to review the mockup that we have added for the mydspace/submission?

Tim Donohue [10:24 AM]
Which mockup are you referring to?

Art Lowel [10:24 AM]
https://wiki.duraspace.org/display/DSPACE/Submission+wizard

[10:24]
I’m also just seeing it now for the first time btw

Tim Donohue [10:25 AM]
Oh, I didn't realize mockups were attached to these pages either. I had glanced at them when they were just text, but hadn't seen the screen mockups yet

[10:25]
This has mockups too now: https://wiki.duraspace.org/display/DSPACE/MyDSpace+as+a+dashboard

Andrea Bollini [10:26 AM]
I have also linked these pages to the use cases as requested by Bram to facilitate the work of the outreach group

Tim Donohue [10:26 AM]
I wonder if it's worth forwarding these along to either #outreach and/or #dcat for their feedback as "users" of DSpace. They look "good" to me at a glance, but would like their input

Art Lowel [10:27 AM]
I’ll go over them in detail by the next meeting

Tim Donohue [10:27 AM]
(Both Outreach and DCAT now have channels here in Slack, see above :point_up: )

Andrea Bollini [10:28 AM]
ok, I will post these links in the two channels to raise the attention. Thanks!

Tim Donohue [10:29 AM]
Sounds great!

[10:29]
Ok, as we are now at the 1/2 way point, let's move this discussion over to #rest-api for REST API updates

Tim Donohue [10:30 AM]
@here: It's now time for DSpace 7 REST API updates, as part of our weekly checkin meeting

[10:30]
@bollini , I know you've been on holiday, but I'm not sure if there's anything specific you'd like to highlight yourself today?

Andrea Bollini [10:31 AM]
I'm only aware about the activities in the REST contract

[10:31]
(discovery) search: https://github.com/DSpace/Rest7Contract/pull/9

[10:32]
authentication: https://github.com/DSpace/Rest7Contract/issues/10

Tim Donohue [10:32 AM]
I believe that is the biggest effort currently, @tom_desair's search endpoint proposal (and I think Tom is on holiday himself this week) https://github.com/DSpace/Rest7Contract/pull/9

Andrea Bollini [10:33 AM]
I think that we almost agree about the discovery search, so I will ask Tom if he want to make the change to the PR and we can proceed to merge

Tim Donohue [10:33 AM]
Sounds good

Andrea Bollini [10:33 AM]
as he will work also on the implementation no need to rush, we can wait for him

[10:34]
authentication is more debate

[10:35]
probably it will be useful to make some experiments. I plan to play a bit with the support provided by Spring Security next week

[10:36]
btw, spring security (should) make easy to implement oauth2/openid or "plan" JWT as well

Tim Donohue [10:36 AM]
Yes, I suspect a basic proof of concept would help alleviate concerns & make it more "tangible". Currently, the idea seems "reasonable" to me, but I'm not sure if it's easily doable with our current AuthN/Z or if it'll result in a major overhaul (the latter of which may affect our development/release schedule)

Andrea Bollini [10:37 AM]
I agree with you, I don't think that we have the resource to rework our AuthN/Z for DSpace 7

[10:38]
so what I propose is *only* to have a standard layer to access our custom / current implementation

Tim Donohue [10:39 AM]
As a related sidenote, I've also been hearing some feedback (from a few folks) that there are users who are concerned about the DSpace 6 to 7 upgrade. I'm starting to feel that the less we change in the data model & configuration the better (as it'd be nice to say "your configs/data easily move forward"), and concentrate more heavily on just the UI

[10:40]
(In other words, we shouldn't strive to do everything at once, as it also makes upgrades that much harder...we should consider keeping major API/data model/config changes for releases post-DSpace 7.)

Andrea Bollini [10:40 AM]
we can provide automatic migration for configuration files where needed

Mark Wood [10:41 AM]
Figuring out how to bring forward our customizations using a completely new framework is going to be daunting enough.

Andrea Bollini [10:41 AM]
this is why I saw that we should provide automatic migration in some case

[10:42]
I don't expect that our item-submission.xml makes any sense in DSpace7

Tim Donohue [10:42 AM]
I'm just saying I've been hearing concerns that we are potentially "pulling the rug out" and making an already hard upgrade much much harder. We need to avoid that perception and/or reality. It's still possible to tweak some configs (and auto migrate them forward), but we shouldn't tweak any configs that aren't "auto-migratable" (in some fashion) (edited)

Andrea Bollini [10:43 AM]
BTW, I expect to hit soon these kind of issues because I need to work on expose the submission configuration over REST

Tim Donohue [10:45 AM]
Right, which is exactly why I bring them up. I fully realize the `item-submission.xml` and `input-forms.xml` are hard to support in that model (and actually may have outlived their welcome anyhow). But, if those configs are no longer used, it's important to find a way to auto-migrate changes folks made there

[10:45]
(and/or find a way to load configs from those files the first time you upgrade...and then ignore them afterwards)

Andrea Bollini [10:45 AM]
ok, we will have more to say on that next week. My expectation is that input-forms.xml will be supported /migrated

[10:46]
item-submission.xml will raise warning that need to be manually solved because it could involve custom classes

Terry Brady [10:47 AM]
Joining late... if auto-migrate is not possible then having a tool to validate compatibility between and old/new version may be useful.

Tim Donohue [10:47 AM]
@bollini: custom code is always an issue, so that's not an unreasonable warning. It's just avoiding warnings on small custom *config* changes I want to see (edited)

Andrea Bollini [10:48 AM]
another thing that we can anticipate so to start to think about.... workflow

[10:48]
right now we have two different workflow engine

Terry Brady [10:48 AM]
It is definitely good to be thinking about this now. Perhaps another page should be added to the contract repo to document which configs will/will not be impacted

Andrea Bollini [10:48 AM]
do we need to support both? do we want to force the migration to the new one in dspace 7?

Mark Wood [10:49 AM]
Dropping the fixed workflow is on the schedule for 7, no?

Terry Brady [10:49 AM]
Since DSpace 6 changed the format of config files from DSpace 5, will we need to ask folks to upgrade to 6 and then to 7, or will we support 4->7 and 5->7 upgrades?

Tim Donohue [10:51 AM]
@terrywbrady : 4->7 and 5->7 upgrades are supported...they just need to realize there's a new config system in place (as of DSpace 6). So the upgrade process is similar to upgrading to 6 now with regards to configs (edited)

[10:52]
With regards to Workflow, I think we can drop the fixed, traditional workflow in 7 if it makes our lives easier. It's already deprecated. So, if we want to simply support Configurable/XML Workflow that's fine.

Mark Wood [10:52 AM]
Isn't that a roadmap item?

Andrea Bollini [10:52 AM]
I'm trying to find the link...

Tim Donohue [10:53 AM]
https://jira.duraspace.org/browse/DS-3041

[10:53]
Yes, it's also listed on https://wiki.duraspace.org/display/DSPACE/RoadMap , but currently under "Post-7.0 Features"

Mark Wood [10:54 AM]
Oh, sorry.

Tim Donohue [10:54 AM]
We can move that up into 7.0 if it makes our lives easier. But, we don't *have* to do so

Terry Brady [10:54 AM]
Would it make sense to advise folks to migrate to XML Workflow in 5/6 before attempting to upgrade to 7? (To simplify the migration process)

Tim Donohue [10:54 AM]
Currently it's not in 7.0 Roadmap as we haven't discussed it in great detail, and what it'd mean to drop Traditional Workflow altogether

Mark Wood [10:54 AM]
Whether you can migrate in 6 depends on what features you use and what UI you use. Each implements something that the other omits.

Andrea Bollini [10:55 AM]
my understanding is that the issues with the xml workflow is that it is not wide used and tested

Tim Donohue [10:55 AM]
@terrywbrady : XML Workflow in DSpace 5/6 is not fully featured. See the subtasks under https://jira.duraspace.org/browse/DS-3041 (XML Workflow doesn't work in JSPUI and doesn't support curation tasks currently) (edited)

Andrea Bollini [10:55 AM]
and it is hard to support because it is only available for the xmlui

[10:56]
but this can be solved in dspace 7

[10:56]
I need to take some time to study in more detail the xml workflow but I guess that it will be easier to support just one in dspace 7

Terry Brady [10:56 AM]
OK... just another tricky consideration for the upgrade

Mark Wood [10:57 AM]
I would love to see the configurable workflow and curation issues sorted. Maybe I can contribute in those areas.

Tim Donohue [10:57 AM]
I think the long term goal here is to have a *single workflow system*. If it makes DSpace 7 easier, we can make this step in DSpace 7. But, we need to (again) ideally ensure an auto-upgrade of workflow configs into DSpace 7

[10:58]
Currently, I think the best candidate for that single workflow system (out of our two options) is the XML Workflow (as it's much more configurable and defaults to same settings as the traditional workflow). But, the final decision need not be that, as long as we have a way to move into it easily from both our existing workflow systems (edited)

Andrea Bollini [10:59 AM]
wait wait :slightly_smiling_face: we don't have other candidate, we cannot develop a new workflow system for dspace 7

Mark Wood [10:59 AM]
Please, no.

Tim Donohue [10:59 AM]
I'm not saying we should. I'm just walking through the logic here. We need *one*, and XMLWorkflow is the best we have. So, we probably should use that as the one (if we are making that step in DSpace 7)

Mark Wood [11:00 AM]
I mean: let's not do a third one.

[11:01]
Configurable workflow needs curation support. Ripping out fixed workflow would make that a lot easier.

Andrea Bollini [11:01 AM]
as you can imagine I'm not a fun of the XML Workflow (as I live happy with the jspui) but it has more (core) feature than JSPUI so yes, also my opinion here is that we should try to put effort in improving it instead to waste time keeping support for two in dspace 7

Tim Donohue [11:02 AM]
yes, I'd rather fix the issues with XML Workflow. And for what it's worth, XML Workflow *didn't* have those major security issues we recently found in Traditional Workflow. So, it's "better" in that way too :wink:

Andrea Bollini [11:02 AM]
for the next meeting or the next I will try to put the contract for the workflow endpoint assuming that we will stay only with the XML Workflow

Tim Donohue [11:04 AM]
Ok, so this has been a great discussion, but I'm realizing we are over time (by 4 minutes)

Andrea Bollini [11:04 AM]
ok, anyone else have plan for the REST API that can be shared?

Tim Donohue [11:05 AM]
Yes, any last updates / questions to share?

Terry Brady [11:05 AM]
I will start a page to document anticipated config changes related to the upgrade and add it to the Contract repo.

Andrea Bollini [11:06 AM]
maybe, a wiki page could be a better location?

[11:06]
it will be easier to move in the documentation and it is not really related to the REST contract

Tim Donohue [11:06 AM]
Yes, config changes can go on the wiki. It doesn't have anything to do with a REST contract

[11:06]
We do have this page where notes could be started (even rough notes): https://wiki.duraspace.org/display/DSPACE/DSpace+Release+7.0+Status

Andrea Bollini [11:07 AM]
anyway thanks @terrywbrady to volunteer for that!

Tim Donohue [11:07 AM]
But, I'd caution against putting things up that are quite uncertain. So, we shouldn't promise something there until we have made a decision on which configs are changing & how they are changing

Terry Brady [11:07 AM]
I can put it where you suggest. If the REST code will be reading the config and acting on it, then it is related to the code

Tim Donohue [11:08 AM]
right, it may be related to REST *code*, but not the contract (the contract defines how the REST endpoints will respond when you query them)

Terry Brady [11:08 AM]
@tdonohue , there where is the appropriate place to document a common understanding of what might need to change

Tim Donohue [11:08 AM]
And technically, REST is likely gonna grab configs via the Java API (so it's likely more related to the Java API anyhow)

Andrea Bollini [11:09 AM]
maybe linked to the notes about the REST API Dev?

Tim Donohue [11:09 AM]
@terrywbrady : if the docs you are talking about are brainstorms/questions...just create a subpage under our https://wiki.duraspace.org/display/DSPACE/DSpace+7+Working+Group

[11:09]
Once the ideas are "finalized" (or nearly final) we can move them over to https://wiki.duraspace.org/display/DSPACE/DSpace+Release+7.0+Status

[11:10]
Ok, with that, let's close up today's meeting. We're already 10 mins over, and I don't want to steal anymore time from you all!

[11:11]
I'll still be hanging around a bit though if further questions/comments come up. Just ping me on one of these channels

Andrea Bollini [11:11 AM]
I agree, put a link also here https://wiki.duraspace.org/display/DSPACE/DSpace+7+REST%3A+Coding+DSpace+Objects as it should be the page where new JAVA developers start from for DSpace 7

[11:12]
ok, thanks to all to join the conversation today! it is mid August!

Tim Donohue [11:12 AM]
Thanks all! We'll talk again next week via Google Hangouts (Thurs, Aug 17 @ 15UTC)

Terry Brady [11:43 AM]
I took a first pass at documenting config file change assumptions and placed it where Tim suggested: https://wiki.duraspace.org/display/DSPACE/Anticipated+Configuration+Changes+in+the+DSpace+7+Upgrade

[11:43]
If this looks useful, feel free to link to other spots.