Virtual CDI meeting 2021-05-26

Agenda (as of email from Arofan to CDI group of 2021-05-26)

1. Admin updates
2. Model and doc generation proposal (Achim)
3. Other issues (if time)

CDI meeting minutes 2021-05-26

Attendees

Achim, Arofan, Dan, Flavio, Hilde, Larry

Apologies

Jay, Wendy

1.      Admin updates

Arofan reported about budget requests for the group.

2.      Model and doc generation proposal (Achim)

In advance of the meeting of 2021-05-19 Achim had provided a presentation that outlines possible solutions for the following modle related topics:

- Identification and reference

- Deep linking

- Modularity

- UML abstractions

- Generic reference

At the meeting, he presented topics from the presentation, starting out by explaining how Eclipse tools can be used to make changes to the model, for example to transform source/target and remove identifiable.

Regarding deep linking Achim suggests to add the UML property isOrdered to all attribute definitions of classes and datatypes, and use the generic selector definition style of the W3C annotation model for references.

Larry and Flavio commented that it would be good to have a case for nested deep linking, as the W3C example has just one level.

Regarding general reference Achim suggested to use the W3C annotation selector for DDI-CDI object attributes, and to include ObjectAttributeSelector that is not defined by the W3C. This procedure could work as a concept that different syntaxes could be translated to.

The approach could be used to reference classes from different modules, where currently only UML associations exist. Using the suggested approach to build independent modules would allow references to other modules as instances. This would be a similar approach as what is used in the model  when associating InformationObject from Data Description with Process. This could be understood as a bridging between those two modules.

Relationship between specifications was then discussed. An issue could be that relationship names like Trace, Refine etc. have limited portability. These relationships are, however, only loosely defined in UML. A discussion followed regarding possible ways to handle this, and different solutions were proposed.

Flavio commented, that human readable diagrams that include similar relationships (Trace, Refine etc.) would be lost when exchanginh the model between canonical XMI and EA. A question is if diagrams containing for example Refine relationships would be used for other than internal purposes.

Achim will look further into solutions for the above topics till next time.

Virtual MRT meeting 2021-05-12

Agenda (as of email from Arofan of 2021-05-12)

1. Update on International Data Week, Seoul, South Korea - https://codata.org/events/conferences/international-data-week-2021/)
2. Issue dispositions and proposals (Hilde/Larry)
3. Budget requests for next year
4. Physical Layout update
5. Planning for IDs/Packaging
6. SDMX Mapping update
7. Other mappings?

1. Virtual MRT meeting minutes 2021-05-12

Attendees: Achim, Arofan, Dan, Flavio, Hilde, Jay, Larry, Wendy

0.      ResourceIdentifier and Collection proposal (Larry)

Larry went through his proposal regarding how to Collection with ResourceIdentifies can be used to reference external metadata objects, for example from Codebook and Lifecycle. According to Larry this could also possibly be used to reference internal CDI classes as user profiles for CDI.

1.      Update on International Data Week, Seoul, South Korea - https://codata.org/events/conferences/international-data-week-2021/)

Based on feedback from Simon a double session is now planned for. Use-cases from external projects will be brought in.

2.      Issue dispositions and proposals (Hilde/Larry)

Larry has submitted solutions in the Jira issues DMT-240, DMT-284 and DMT-315. The MRT members were asked by Arofan to comment on them till next time.

Flavio reports that CDI-63 is innocuous and can be closed.

3.      Budget requests for next year (deadline May 24^th)

Achim proposed for MRT to send in a provisional request for funding of a DDI-CDI Dagstuhl hybrid workshop, for Europeans to take part in the physical event.

Arofan additionally suggested to apply for funding for an MRT Sprint meeting next April/May.

Both proposals were agreed by the MRT group. Both are dependent on COVID related travel possibilities, as well as travel allowance from the participating organisations.

Given that the MRT has been approved as an official working group by the Scientific Board, Arofan suggested we need to find a new name for the group and asked people to think about proposals till next time.

4.      Physical Layout update

Larry presented a new proposal for PhysicalInstance (included in Jira issue CDI-31). This includes a simplification of how formatting relates to DataPoints. As ValueMappings are about strings, the current ValueMappings could be changed to ‘StringFormat’. MRT members are asked to look at Larry’s proposal till next time. 

5.      Planning for IDs/Packaging

Achim will send out a proposal regarding identification with enhanced documentation till next time.

Achim is also working on a complete framework that can provide basis for what is possible to do in relation to modularity. He will present some analyses of the model at the next meeting.

6.      SDMX Mapping update

Flavio has been working on the dimensional model, hierarchical code set and attributes. He will present more about the SDMX mapping at the next MRT meeting.

7.      Other mappings?

Larry is continuing on the Codebook 2.5 mapping. He can proceed when the Collection proposal is in place. Arofan pointed to the need to narrow the semantics regarding what the relationships point to. Dan mentioned the importance of keeping restrictions due to interoperability at the metadata level, while Flavio means that an extension mechanism in CDI that could be part of the model, is needed. Arofan proposed to look into what can be done on the level of implementation guidelines, for example guidelines for extending from CDI to SKOS. Achim, Arofan and Flavio will meet before next time to discuss this further.

Achim, Arofan and Hilde will meet to discuss further Jira issues.

Virtual MRT meeting 2021-05-05

Agenda (as of email from Arofan of 2021-05-05)

1. DataSciCon presentations? (International Data Week, Seoul, South Korea - https://codata.org/events/conferences/international-data-week-2021/)
2. Webinar series - presentation on DDI-CDI
3. Issue dispositions (Hilde)
4. Study-level mappings (Larry)
5. Maintenance planning (Arofan/Wendy)
6. SDMX Mapping update
7. Use Case update (Flavio/Arofan/Achim)
8. Physical layout update

Virtual MRT meeting minutes, 2021-05-05

Attendees

Arofan, Dan, Flavio, Jay, Hilde, Larry, Wendy

Apologies

Achim

1.      DataSciCon presentations? (International Data Week, Seoul, South Korea - https://codata.org/events/conferences/international-data-week-2021/)

The International Data Week in Seoul will be half-virtual. Simon is organizer. A placeholder for a DDI-CDI session has been put in. Ideas for content related to topics around ‘Cross Domain Integration – lessons learned’ was presented by Arofan. Flavio commented that use-cases need to be provided for all of the topics. The programme will be further developed and a draft will be sent to Simon.

2.      Webinar series - presentation on DDI-CDI

The first webinar in the series of the Training Opportunities Group: ‘Introduction to Metadata for Research Data Management: A Data Documentation Initiative (DDI) Perspective’ was a success, and a report from the webinar was sent to the Scientific Board, the Executive Board and Marketing.

The next webinar will be related to DDI and FAIR. A webinar on DDI-CDI is planned for around it’s release. Dan suggested to make this problem centric to show what CDI can do. Flavio suggested to bring in other standards as well. Like what can DDI- CDI do together with Codebook and Lifecycle, using an example.

4. Study-level mappings (Larry)

Larry has been working on mapping to Codebook 2.5. For CDI to relate to StudyDescription, for example, additions to the CDI model are needed. Larry’s proposal is about defining a call that can reference a set of URIs that links to metadata objects. The links could be to PROV objects, Lifecycle elements etc. According to Larry this can be done by extending Collection a little, which should not be hard to do.

Dan pointed out that regarding semantic interoperability problems there are several ways to solve them, and that Larry’s proposal seems reasonable. Arofan pointed at the need to organize things so that there will be only one way to do this.

Jay added that JSON-LD does something similar. A concept with multiple URIs can be added, referring to representations of the same concept in different locations. Flavio pointed out that the URIs could point to different things than concepts.

Jay suggests to look into how ‘arrays of arrays’ are done in JSON-LD. Larry suggests CDI could do ‘collections of collections’.

While the MRT members were positive to Larry’s approach, a missing piece is according to Arofan some primary semantics of what the links are to. Achim, Flavio, Larry and Jay will meet later to look at this. Larry will make some diagrams describing the proposed solution.

5. Maintenance planning (Arofan/Wendy)

When the CDI is published we need to make sure to keep it in line with product maintenance processes. Achim has been working on the sub-set of UML features that CDI uses. This has shown great interest in the UML community. Achim will provide a document and send this to the TC.

The issue of automating the CDI production process was raised. Currently this is done on a manual basis by a few core individuals. Issues around the applicability of the COGS system for publication puropses were raised. Larry pointed out that COGS cannot represent cardinalities that CDI has. It would further need to be checked if COGS would be able to handle the components of Achim’s UML profile. Steps here would be that MRT sends a description about what they do to the TC, keeping in mind that all product teams do not need to use the same production system.

Agenda points 3, 6 -8 were postponed till the next time.

Virtual MRT meeting 2021-04-28

Agenda (as of email to the MRT group of 2021-04-28)

1. Updates on citation mappings (Achim)
2. Issues overview and progress (Hilde/Arofan)
3. SDMX mapping (Flavio)
4. Other mappings?
5. Training Webinar series suggestion/discussion re: CDI topic

MRT meeting minutes 2021-04-28

Attendees

Achim, Arofan, Dan, Flavio, Hilde, Larry, Wendy

Apologies

Jay, Oliver

1.      Updates on citation mappings (Achim)

Work has started and progress updates will follow later on.

2.      Issues overview and progress (Hilde/Arofan)

Hilde and Arofan have been looking at Jira issues that have been classified by a group led by Larry. Some have been closed, and others will be assigned to members of the MRT group to find a solution for them. Hilde and Arofan will write emails to the MRT members to assign tasks related to this.

3.      SDMX mapping (Flavio)

Flavio is working on a mapping between DDI-CDI and SDMX and had a meeting with Achim and Arofan to discuss.

The work on the mapping to the current version of SDMX goes well, and about half of the work has been completed. Some collaboration with people in the SDMX group would be useful, and Jared will be contacted by the MRT group to write to SDMX regarding this.

Possibilities for a formal collaboration between the DDI Alliance and SDMX bodies could be a discussion point for the Scientific Board, after the work on the Scientific Plan has been completed.

Flavio mentioned that steps for collaboration should be made at the latest during the next autumn.

4.      Other mappings?

Larry is working on a mapping between DDI-CDI and DDI-Codebook version 2.5. Comments provided by Larry in connection with this work is that directionality of model arrows makes mappings easier and that ValueMapping on a DataPoint level as in the DDI-CDI rather than at an InstanceVariable level makes the mapping more complex. Would Collections of DataPoints as an attachment point for ValueMapping be a solution? The Formats working group will look further into this, and will seek to find a solution that would work for all the different data structures that DDI-CDI can describe.

Achim and Arofan are been working on use-cases about integration of datasets of different types, related to the role of DDI-CDI in EOSC report, that could form use-cases for the DDI-CDI specification. Flavio is working on a use-case together with George Alter and others.

5.      Training Webinar series suggestion/discussion re: CDI topic

Arofan explained about the upcoming webinar series of the Training Opportunities Group, where DDI-CDI is mentioned as a one of the topics for the webinar programme.

The members of the MRT were asked to provide input on which topics would be relevant to present about DDI-CDI in connection with the webinar.

The following themes were mentioned:

Use-cases

• Transformations between data structures
• How to integrate data across domains using DDI-CDI
• Provenance and lineage
• How to deal with the complexity when using DDI-CDI in practice

The group was also asked about ideas for further ideas for topics for the webinar series, DDI and FAIR being the next topic to be presented.

Wendy mentioned DDI-Lifecycle 3.3 and serialization as a topic.

Flavio mentioned use-cases and limitations of the different standards in the DDI product suit, and how the different specifications usefully can be used together.

Virtual MRT meeting 2021-04-14

Agenda (as of email from Arofan of 2021-04-14

1. Updates on Annotations subgroup
2. Issues and status
3. Organization for next steps

MRT meeting minutes 2020-04-14

Attendees

Achim, Arofan, Flavio, Hilde, Jay, Larry

Apologies

Dan, Oliver, Wendy

1.      Updates on Annotations subgroup

The group will stop meeting until the modularization issues including packaging and layering discussions are done.

Achim is currently working on a mapping of annotation content to Dublin Core.

2.      Issues and status

Larry’s group has been categorizing issues as to whether they can be closed or need to be done or discussed before the public release.

As a next step Arofan and Hilde will go through the issues for discussion and ask people to prepare proposals with pros and cons for those that are important for the release.

3.      Organization for next steps

The goal is to have a draft ready by the end of June.

The preparatory work will be done in two rounds. The things that are likely to impact the model most will be handled first. Work in three areas will start now:

• Identification, linking, deep linking: Achim has provided a proposal regarding identification. He will widen his current proposal to include all content touching on identification and reference, and will provide widened draft by the end of April.
• Modularity: Flavio, Achim and Larry will look into this and provide a draft with pros and cons. The work is estimated to take place in May.
• Formats: Flavio and Larry will look into this, and make a statement regarding what CDI will support and what it will not support. An analyses of use will be provided in 2-3 weeks. Some guidance will also be provided. Possible changes can be thought of in the second round.

In addition to this, Hilde will start to look into updates the field level documentation, with help of Dan and Wendy. Larry will help with analyses of completeness and structure of the field level documentation.

Other topics mentioned for the second step were:

• High level examples for mapping to other specifications of importance to the release.
  • DDI-C, DDI-L
  • SDMX, DCAT
• Use-cases originating from Arofan, Achim and Simon’s work on the DDI-CDI for EOSC report.
• Process (Jay and Flavio). This needs more documentation. Flavio pointed additionally at the need to compare the way CDI solves things, to how other projects think, for example the UNECE process discussions. This could for example be included in an implementation guide to be made available post-release.
• How to connect concepts to vocabularies. Look into how Lifecycle deals with this (Jay, Dan). A question is whether this is doable for the first release. This needs to be checked out.
• Look into if the collection pattern can be amended to point to external resources (Larry).
• Outcome of proposals for solving Jira issues of importance for the release.

Next meeting will focus on the timeline for the release work

Virtual MRT meeting 2021-04-07

Agenda (as of email from Arofan of 2021-04-06)

1. Updates on Annotations subgroup
2. Issues and status
3. Documentation and examples (brainstorm)

Virtual MRT meeting minutes 2021-04-07

Attendees

Achim, Arofan, Dan, Flavio, Hilde, Jay, Larry, Wendy

Apologies

Oliver

General updates

Arofan reported that DDI will be presented at Dublin Core meeting.

Topics and presenters:

• Overview: Jared
• Codebook: Jane
• Lifecycle: Barry
• CDI: Arofan

1.      Updates on Annotations subgroup

This group has not met since the last MRT meeting, as the date fell on Easter Monday.

2.      Issues and status

The group led by Larry had made good progress on categorisation of Jira issues for discussion since the former MRT meeting. A draft has been available to the group post meeting, for discussion at the MRT meeting on April 14^th.

3.      Documentation and examples (brainstorm)

Topics that will be in focus in the coming time:

• Task list and work processes
• Field level documentation
• Comments on the spec
• Formalise standards alignment
• Examples (from webinars; worked examples)

A brainstorm regarding examples followed, and the following ideas were mentioned:

• DCAT AP/CDI example (Arofan, Achim, DCAT).
• Mapping to other standards (SDMX, Lifecycle, Codebook).
• How to use CDI together with vocabularies from other standards, for example SKOS (Jay, Flavio).
• FIP example.
• Process examples (Jay). Make these more valuable to people. What can be done without steps for example. Parameterized vs. non parameterized processes, etc.
• Transformation between data structures (Larry, Flavio). Include SDTL?
• Example from the CDI-EOSC report on cross-domain CDI usage (Arofan, Achim)?
• Possible BLS usecase (Dan).

Arofan pointed out that people now want small standard components for use together, rather than monolithic standards, for example DCAT + CDI Variable Cascade. This could be viewed as a design principle. Flavio commented to this that the model needs to be looked into to find ‘bridge classes’ to other standards. ResourceIdentifier as a possible linking point was mentioned. Achim pointed to the need to look into a UML way to refer to other specifications or model parts,  and explore what is best at each level of abstraction etc. This topic will require a separate discussion.

Arofan also pointed to the importance of user guides to inform the understanding and implementation of the examples.

Forthcoming work

• Organizing the work will be a topic next time.
• Packaging and formatting has already been agreed as topics to be discussed in separate groups.
• Larry will present the outcome of the categorization of Jira issues for discussion.
• The TC will receive a list of planned changes to the model when those have been agreed. This is needed to decide whether a second review of CDI is needed before publication.

Virtual MRT meeting 2021-03-31

Agenda (as of email from Arofan of 2021-03-31)

1. Updates on Annotations subgroup
2. Issues and status
3. IASSIST and other outreach topics
4. Possible subgroups for further work (e.g., Packaging subgroup? Others?)

Meeting minutes, virtual MRT meeting 2021-03-31

Attendees

Achim, Arofan, Dan, Flavio, Hilde, Larry, Wendy

Apologies

Jay, Oliver

0.      General info

Arofan, Achim and Simon are working to finalize the DDI-CDI for EOSC.

Flavio has started to look into relationships between DDI-CDI and SDMX.

1.      Updates on Annotations subgroup

The latest discussions have been centered around packaging and modularity, as well as mapping to citation standards (Dublin Core/DCMI).

The group has also been looking into how date can be simplified for Annotation, and, possibly, for other aspects of the model where interchange with other standards is important. The broader topic of rules for interchange has also been touched on.

2.      Issues and status

A group is currently working to classify DMT and CDI issues in Jira into topic groups for discussion. The group is continuing to work regularly on this.

3. IASSIST and other outreach topics

Larry and Hilde got an abstract about Data Description in the DDI-CDI accepted for the virtual IASSIST conference 2021. They are interested in input from the group regarding the content of the presentation. Arofan pointed to the importance of explaining how CDI can be used to handle new types of data. Examples from EOSC project that Arofan and Achim is working on can be used for this. Flavio pointed at provenance and lineage relationships between datasets as important topics. Dan mentioned that a description of the process model could be useful to include.

As this is just a presentation of normal length it was agreed that the main focus will be on data description and examples, and that process could be covered by one slide. Arofan and Achim will provide the EOSC examples for the presentation.

3.      Possible subgroups for further work (e.g., Packaging subgroup? Others?)

Packaging will require a separate sub-group. Based on interest from DCAT and other, Achim suggested the Variable Cascade to be worked on as a first example. The logical rather than the physical aspects will be in focus of the work.

Flavio is pointing out that packaging is one among other important criteria, and an approach to the complexity is needed. Achim agrees with this, and suggests that the model is looked into, as to which parts of the model can be made modular without large efforts, and of what needs to be done. The goal is to make improvements regarding modularity for the first release and continue to work on this. Larry and Achim are currently carrying out analyses on the interconnectivity of the model.

Formatting is a different topic that Flavio proposed to focus on. Does CDI covers the necessary formats? A separate group will be convened to work on this on the immediate term, while the packaging work will start later.

Focus will also need to be put on mappings, to SDMX (ongoing), and to Codebook and Lifecycle, prior to the release.

Virtual MRT meeting 2021-03-24

Agenda (as of email from Arofan 2021-03-23)

1. Updates regarding SDTL, Dataverse, and EOSC (plus any other short news)
2. Annotations proposal from sub-group & next steps
3. Combined issues list/updates
4. Model development collaboration: "sandbox" ideas, etc.

Meeting minutes Virtual MRT 2021-03-24

Attendees

Achim, Arofan, Dan, Hilde, Larry, Wendy

Apologies

Flavio, Jay, Oliver

1.      Updates regarding SDTL, Dataverse, and EOSC (plus any other short news)

Arofan reported that good contacts for possible CDI implementation have been made through the CDI webinar and symposium work, as well as through the EOSC/DDI-CDI project that Simon, Arofan and Achim are running. One of those collaborations are Dataverse, where how to express codebook information and variable level metadata in CDI is currently explored.

Another topic of interest that has arisen through the contact with SDTL is to use the CDI Process model to drive business processes, in addition to documenting them.

2.      Annotations proposal from sub-group & next steps

Larry has sent out a proposal for changes to Annotation prior to the meeting, which includes a proposal to break up the Annotation class. The proposal contains, among others, the idea to include a Note class, that would allow to attach a note of controlled vocabulary type, or any information element with a URI, to any CDI element. In the proposal Note can use the data type ResourceIdentifier can be used to point to things without object type constraints. The strength of this approach is the freedom it gives to reference anything. However, to use associations would however be required when control over the type of the referenced object is needed.

Arofan suggests that the proposal to break up the Annotation class would not represent huge, but not insignificant changes to the model. To make the class more detailed would on the other hand require less work.

The proposal from the Annotation group received overall positive response from the participants at the meeting, but will need to be reviewed by more group members and in relation to mapping to other standards use cases before approval.

A new sub-group will look into the model packaging and what can be decoupled. This group will collaborate with the Annotation group.

3.      Combined issues list/updates

A separate sub-group is looking into this. It is currently working on grouping DMT and CDI Jira issues that need discussion, and will get back to the full MRT group with a proposal.

4.      Model development collaboration: "sandbox" ideas, etc.

How to bring in changes or new content to the model was discussed. While collaborative modelling platforms were mentioned, Achim pointed out that it is again important to have controlled procedures for implementing changes and new content in the model. People can work on model parts in sandboxes that can be copied into the model by a model master.

Flavio will again be asked by Arofan if he would be willing to take this role.

Regarding the XMI creation from EA, Achim explains that this is currently done manually in 5 steps. Achim will provide a description of the procedures he uses for this. He is also exploring how Eclipse can be used for XMI creation purposes.

Virtual MRT meeting 2021-03-17

Agenda (as of email from Arofan of 2021-03-16)

The focus this week will be on the bug disposition/discussion plans from the meeting Larry organized today (Tuesday 16^th) - he sent an email around earlier for people to look at.

Meeting minutes virtual MRT meeting 2021-03-17

Attendees

Achim, Arofan, Dan, Hilde, Larry

Apologies

Flavio, Jay, Oliver, Wendy

1)      Report from the Annotation group

The current CDI packaging is more conceptually based. Work is ongoing to explore how the model can be modularized for usage of separate modules in different contexts.

Larry had sent out a couple of Jupyter dendrograms to the groups, based on the associations in the model. These show how the classes in the model group together based on common associations. The dendograms provides input to the discussion on how future the future packaging could look like, from one specific perspective.

A different perspective would be the one of use-cases. A common use-case in the EOSC work of Achim, Arofan and Simon is how  DDI-C examples would look like in CDI, and look at clusters this perspective. This relates to the work of Larry and Achim about DDI2 in R.

Related topics were discussed, among those:

• Leave out inheritance super-classes from implementation guidelines
• Flattening of the level of the model
• Quality of associations: What is necessary and what could be solved differently?
• Deep linking
• Size of profiles: How small can they be?
• How can the model be more easy to adapt, for example for JSON developers?

The Annotation group meets again on 2021-03-22 to discuss further.

2)      DMT Jira issues discussion

A smaller group is looking into this. On a meeting on 2021-03-16 a set of pre-Berlin Sprint DMT Jira issues were classified according to if they could be ‘closed’, ‘postponed’ or ‘needs further discussion’. The classification was based on comments from the creators of each of the issues, as well as a review performed by Larry. The classification of the pre-Berlin Sprint DMT Jira issues made was agreed at the MRT meeting.

The next step for Larry and the smaller group will be to go through the DMT issues that were fielded during the MRT period, as well as the CDI post the review release issues, and classify them in the same way as the pre-Berlin Sprint ones.

When all of the open issues have been reviewed, a groping of them will be made, and finally a priority list for issues solving will be made by the full MRT.

Provided by Larry for the next meeting in the smaller group, which will take place on 2021-03-23: Updates on classification of DMT issues fielded in the MRT period, but before the review release.

Virtual MRT meeting 2021-03-10

Agenda as of 2021-03-10

(1) Annotations sub-group update (and related)

(2) DCAT mapping update

(3) SDMX mapping planning

(4) Modular design/packaging

MRT meeting minutes 2021-03-10

Attendees

Achim, Arofan, Dan, Flavio, Hilde, Jay, Larry, Wendy

Apologies

Oliver

(1)   Annotations sub-group update (and related)

Achim reported from discussions in the Annotation group.

• How to make the module more modular was discussed. Larry is performing cluster analyses to check connections between clusters. Achim will analyse associations to find out where deep/light connections are.

It is an agreement in the group that it would be nice to have some parts of the model as a separate pack, for use in different contexts. The Variable Cascade was mentioned as one possible example.

Documentation should be developed that reflects CDI usages for specific purposes. Currently an experienced user must understand the full model in order to use a part.

Achim suggests to use Package in UML for this.

Larry suggested to use the collection pattern to create profiles. This would allow users to create own collections.

Annotation as an overlay has been discussed, where a connection could be made that would not require remodeling. How to link to other standards also formed part of the discussion. Larry pointed out that Annotation currently has an issue in that a relationship to an agent requires Agent to be a class. An ExternalControlledVocabulary could be used alternatively, to refer to a DDI or PROV class, for example. Annotation will be remodeled when issues around it are sorted out.

• Wendy asked how much content is allready in the published review version of the model, compared to what is expected for the first version.

Arofan then replied that there will not be big changes in the specification, but that forthcoming changes in the packaging is expected that regards grouping more than content, and that alignment with other specifications needs to be addressed.

(2)   DCAT mapping update

Achim and Arofan is currently working on a project with Simon that regards what DDI-CDI can do for EOSC, focusing on EOSC requirements. One of the focuses is to find out how DDI-CDI can be used in connection with DCAT. A DCAT expert is working on this in collaboration with the group. The outcome of the DDI-CDI EOSC project will be a report that can feed into DDI-CDI.

Achim explained that the DCAT/CDI collaboration project is about how the standards can be used together, rather than about mapping.

Flavio suggested that DCAT AP also is looked into in this connection, as this is used content of importance to statistical agencies.

A discussion about linking to other standards, as well as of the notion of attachment and attachment points then followed.

Achim mentioned that in relation to DCAT there are two perspectives: The CDI perspective and the DCAT perspective. Each perspective could contribute something. It would be good if DCAT could refer to a rich metadata instance, that could be CDI. Flavio and Dan pointed out that what DCAT has done and what it needs would be useful to explore. The collaboration with the DCAT expert continues, and a report is expected end of March. Discussions around the topic involving MRT and possibly external experts will then follow.

(3)   SDMX mapping planning

Flavio is looking into possibilities for this, and will check out how other standards deal with it.

Arofan points out that the importance to have an official recognition  between the institutions of the purpose first, and then to develop a timeframe for the work.

(4)   Modular design/packaging

Achim points out that ongoing analyses will help to understand the extent of the issue, how it can be fixed, and how much work it will take to do the fixes. Results will form basis for deciding what can be done prior to- and after the first publication. Possible remaining work could be announced as a task for a second CDI release. Achim recommends a pragmatic approach to this, as to what can be done without large efforts, and what would require more time. Improvement of the packaging for a more flexible usage should be accompanied by implementation guidelines for main purpose usages, as well as for usages in other contexts. The connections between packages should be minimized and the UML Package structure will be used for this.

Larry pointed again to the idea of machine actionable packaging. This could be considered as a second, post release, step.

The group will look into how modularization is done in specifications that has solved modularisation well, for example schema.org.

The suggested approach was agreed by the group.

Any other business

• Larry suggested that the group discuss content for his IASSIST presentation in a specific MRT meeting. 31^st of March was agreed as a date to discuss this and other possibilities for presenting DDI, possibly in collaboration with the Training Opportunities group.
• The next MRT meeting on March 18^th will focus on Jira issues.

Virtual MRT meeting 2021-02-10

Agenda

1. Report from Dimensions group and possible follow-on activities with SDMX mapping
2. Report from the Annotations sub-group meeting 
3. Issues overview from Larry/Hilde (review and process for dealing with these)

Minutes virtual MRT meeting 2021-02-10

 Attendees

Achim, Arofan, Dan, Flavio, Hilde, Jay, Larry

Apologies

Wendy, Oliver

Administrative

Achim provided a short report from the first meeting of the Scientific Board. Minutes from that meeting is found here .

As the new SB will call for an update from the current work and future plans of each of the working groups of the Alliance, the MRT will arrange a separate meeting for this on Wednesday February 17^th. Larry created an issue in Jira (CDI-33) where group members can enter ideas for this.

1.      Report from Dimensions group and possible follow-on activities with SDMX mapping

Dan reported from this group: The CDI spec has more classes than GSIM. This is however expected for a logical model closer to the implementation level. It would also be possible to implement CDI without using all of the classes. Examples for this are needed.

Flavio will look into possibilities for a mapping between DDI-CDI and SDMX (that is currently moving into microdata). This could be a project for the future (after the first publication of DDI-CDI?).

2.      Report from the Annotations sub-group meeting 

Larry reported from this group: AnnotatedIdentifiable expands content of Identifiable to include an optional annotation, thus currently uses Identification by inheritance. This information could alternatively be a set of properties. Annotation has some semantics in the name relating to the term ‘Note’ that may be confusing as the core object of this class is to cover discovery information.

AnnotatedIdentifiable could be broken up in persistent and changeable parts and should be able to point to things. Achim explained that the W3C annotation framework is using “selectors” to address parts of an identified object. See CDI-4 by Achim for more on this discussion. Achim added to this that if a similar approach is used this could be used for every specification (CDI, Codebook, Lifecycle).

3.      Issues overview from Larry/Hilde (review and process for dealing with these)

Larry has been reviewing all DMT issues marked as ‘resolved’. He has classified these issues as to whether he feels they should be re-opened or could remain closed. He sent out a spreadsheet post-meeting including his proposals for the group to look at. This will be adressed at the MRT meeting on February 24^th. Many of these issues are MRT tasks created in connection with Sprints etc.. A question is how to handle such process related tasks in the future.

Hilde has send out a workbook that contains a set of older pre-Berlin Sprint issues and asked each of the issue creators to review them.

Virtual MRT meeting 2021-01-27

Agenda (as of email from Arofan of 2021.01.27)

1. Quick update on discussion with Training Outreach (following on from Webinars)
2. Report from side-meeting on issues resolution (Larry?)
3. Organization of sub-groups
   1. Sub-groups
   2. Process
4. Examples and high-level documentation (and other documentation?)
5. Timing and game plan
6. Other areas (if not touched on in 1 above, from the list of areas we summarized last week)

Minutes, virtual MRT meeting 2021.01.27

Attendees

Achim, Arofan, Dan, Hilde, Larry, Wendy

Apologies

Flavio, Jay, Oliver

Admin

Arofan informed about the following:

• There will be regular MRT meetings at the usual time (Wednesdays at 10:00 Eastern/16:00 CET). Next week’s MRT meeting will be cancelled, as the first meeting of the new SB is scheduled for the usual MRT time. Achim and Hilde will make sure that a different timing will be chosen for forthcoming SB meetings.
• Jared is organizing a training session on DDI for the Dublin Core Metadata Initiative. Instructors will be Jared (introduction), Barry (Lifecycle), Jane (Codebook) and Arofan (CDI). Achim presented DDI at a DC conference in 2008, which Arofan will reference in his talk.

1.      Quick update on discussion with Training Outreach (following on from Webinars)

Based on the experiences from the CDI webinars held in collaboration with CODATA and the training sessions held at EDDI and the FAIR Convergence Symposium in December, a series of web-based training events is under planning by a collaboration between the CDI webinar team and the Training Opportunities group. CODATA has offered to use their webinar platform and to help with administrative tasks, which we view as a highly appreciated contribution from our new member of the DDI Alliance. A program that includes a series of webinar on different topics is under planning. The plan is to have this sent out to the MRT group, the new SB, the Marketing team and the Training group, before the next meeting of the Traing group, that will take place on February 2^nd.

2.      Report from side-meeting on issues resolution (Larry)

A small group consisting of Achim, Dan, George, Hilde, Jay and Larry met on Tuesday January 26^th to discuss resolution of Jira issues. Larry sent minutes to the srg list after the meeting.

a. Organization of sub-groups

Larry reported that small groups have been formed at the meeting on the 26^th to find solutions some of the important issues:

• Identification (CDI-3): Achim will ask Flavio to work together on this. There is agreement about a solution, but an efficient implementation needs to be discussed.
• Annotation (CDI-4): Achim, Dan, Jay and Larry signed up for this group. Achim has provided a proposal for solution. Arofan has been asked to convene the group. Jay will contribute in relation to schema.org.
• Dimensions (CDI-16): Dan is responsible for this. Arofan will be in the group and Flavio will be asked to contribute as well.
• Issues resolution follow up group: Larry and Hilde will follow up on this by categorizing issues and propose priorities. Wendy volunteered to help regarding older issues that are not closed, dating before the MRT period.

b. Process

At the meeting on 26th, George proposed a consensus approach for making agreements regarding issues resolution: Each sub-group prepares proposals to which the members of the larger group is asked to indicate their consent, and where no comment is interpreted as an agreement. This procedure was tentatively agreed at the MRT meeting.

3.      Examples and high-level documentation (and other documentation?)

a. Examples

Arofan mentioned the importance of having worked examples that explores areas of the spec, as well as the syntax representations. This could work as a sanity check of the model.

Possible examples that mentioned were:

• Australian Election Study
• Units of measure
• org/DCAT mapping (person outside MRT)
• JSON-LD (person(s) outside MRT)
• Usage of CDI together with Lifecycle and Codebook
• Possible examples emerging from the EOSC DDI-CDI project run by Simon, Achim and Arofan

Examples could also be used as a basis for future training material.

A question is which of the examples could be automated rather than being made by hand.

b. Documentation

The high level documentation (specification) can be finalized when agreed changes on the model have been implemented. Larry explained that new diagrams needs to be produced when the changes the model have been made. Arofan pointed out that the naming/use of terms in the model might need a review. This has to do with the way we communicate about the model. Dan means, on the other hand, that the current model uses precise terms.

Arofan will be responsible for the high level documentation, and will provide an overview of which chapters of the spec that is likely to change most.

Hilde will look into the field level documentation together with others in the MRT group, prior to the release.

4.       Timing and game plan

The sub-groups as specified under point 2 will meet in the coming days to start their work and will report back to the larger MRT group at the next meeting.

Larry will start to look into the Australian Election Study examples as well as the blood pressure example. A goal is to produce an R package for DDI-CDI.

At the meeting on the 26^th Larry explained about an agreed association that had disappeared from the model. To be able to detect similar deviances before the release, Achim will look into possibilities to produce a diff programme that could document changes to the model between versions.

Virtual MRT meeting 2020.01.21

Agenda (as of email from Arofan of 2021.01.19)

1. Summary of review inputs

    - Issue tracking

    - Webinar inputs

2. Topics for revision work

a) Mappings to other DDI versions/worked example

b) Process model and impact of recent discussions (Convergence Symposium)

c) Discussions with GO FAIR (FIPs, FAIR Data Objects/Points)

d) Representation (XML? RDF?)

e) DCAT mapping

f) Annotation and Citation (including access, licensing, notes, etc.)

g) Other open issues (IDs, string clean-up/rationalization, etc.)

h) Documentation work

i) Input from EOSC project

3. Scope and Timeline

4. Additional people

Minutes, virtual MRT meeting 2020.01.20

Attendees

Achim, Arofan, Dan, Flavio, Hilde, Jay, Larry, Wendy

Apologies

Oliver, Simon

1.      Summary of review inputs

The main topic of the meeting was to look into the range of work, as well as the scope and timeline for the publication of the CDI.

    - Issue tracking

See point 2 below.

    - Webinar inputs

People from the MRT group held last autumn a series of CDI focused webinars in collaboration with CODATA that received lots of attention. We took part in activities of the FAIR convergence Symposium and taught DDI-CDI at EDDI. This resulted in interest in DDI/CDI and possibilities for further collaboration.

Achim raised the question about what to do about CDI webinars that were planned but have not yet taken place. We need to decide which of those to follow up on. Larry pointed out we need to distinguish between which ones would be needed for the first publication (areas where we need input) and which ones can wait till after.

A discussion followed about the Upper model (Jay) about a possible extension to handle infrastructures around processing. Jay will pursue this, and follow up with George Alter to discuss the possibility of taking this on board. That is however not likely to happen for the first release of the CDI.

2. Topics for revision work

a)      Mappings to other DDI versions/worked example

A worked example about how CDI and Lifecycle can work together should be planned for.

b)      Process model and impact of recent discussions (Convergence Symposium)

Possible bugs need to be addressed.

c)       Discussions with GO FAIR (FIPs, FAIR Data Objects/Points)

Arofan, Achim and Hilde have been in contact with Erik Schultes and Barbara Magagna from the FAIR team regarding how FIPs etc. technically fits with CDI, and how CDI possibly could be used in their ecosystem. A meeting has been agreed but has not yet taken place.

d)      Representation (XML? RDF?)

Representations need more focus. Would a webinar (as planned for earlier) be a good way to move forward with this?

e)       DCAT mapping

Alejandra Gonzales Beltran is working on a mapping between CDI and the latest version of DCAT. Flavio who has experience of this from similar work at StatCan will look into this when final.

f)        Annotation and Citation (including access, licensing, notes, etc.)

Needs to be looked further into.

g)      Other open issues (IDs, string clean-up/rationalization, etc.)

Jira issues fielded while the review process so far need to be looked into. (Available at https://ddi-alliance.atlassian.net/wiki/spaces/DDI4/pages/897155138/DDI-CDI+Comments).

h)      Documentation work

Cleanup of the model documentation needs to be performed prior to the publication.

i)        Input from EOSC project

CODATA is responsible. Achim and Arofan are contributing from the MRT/CDI side. The project is looking into how DDI-CDI can be used in a data sharing environment. Output from the project will be formalized as requirements that will be fed back to the MRT team. The project plans to deliver in April 2021.

4.      Scope and Timeline

• Wendy laid out the timeline needed for the publication process. The voting requires one month. Then the TC will need a week or two to prepare for the publication. If substantial changes made are an additional review of those changes will need four to six weeks.
• The question is then whether to publish in the shorter term (Q2), and get back with an update not so long after, or wait a little longer (Q3) and possibly bring in more content. This will needed to be agreed in the short term.
• Communication with the TC regarding decisions about how to proceed is important.

5.      Additional people

This point was not addressed.

Action points

• Weekly MRT meetings will again be held.
• Discuss which CDI webinars to follow up on.
• Decide regarding scope of work and publication.
• Larry will convene a group that will look into how the issues raised can be fixed and the process to implement the fixes and improvements (Achim, Flavio, Hilde, Larry, possibly Dan, other interested?).

Virtual MRT meeting 2020-04-29

Agenda

Identify target groups for review webinars on basis of spreadsheets and input from Simon Hodson of CODATA regarding break-down of the scientific unions and relevant standards.

MRT meeting minutes 2020-04-29

Attendees

Achim, Arofan, Dan G., Hilde, Jay, Larry, Oliver

Apologies

Flavio, Wendy

Discussions with CODATA

Arofan reported about discussions between Achim, Simon H. and himself regarding review of the DDI-CDI. Simon H. is very supportive of the DDI-CDI, and is willing to help promoting the review. He has provided an email list of engaged people involved in standards work in different domains within the RDA, and is willing to help further.

Webinar planning

The next question is how to group interested reviewers in topic groups for the webinars under planning. We are considering 6 to max. 12 webinars to be held in connection with the review.

Arofan proposed for the weekly MRT meeting in the forthcoming time to replace the weekly MRT meetings. All MRT members would be welcome to attend the webinars to be organised for each topic group of possible reviewers, and after each there will be a 15 minutes MRT discussion.

This was agreed by the group.

A discussion then followed regarding which topic groups could be split up or collapsed, looking at the MRT spreadsheet of contacts by topic, and Simon H.’s list.

Geoscience data was considered as a candidate for a stand-alone webinar, while life sciences for example, could be a candidate to merge? Larry introduced the idea to organize webinars according to topics that could be interesting for people from different domains, like for example handling of binary formats/streaming.

Arofan then proposed to start the webinar series with some clearly defined topic groups, like official statistics and health sciences and learn from that to plan forward.

Next MRT meeting and follow-up work

Arofan proposed to hold the MRT meeting next week as usual.

Side meetings will be organized in order to reorder the topic groups and assign people to relevant webinars, as well as to start to prepare material for the presentations.

Virtual MRT meeting 2020-04-22

Agenda (as of email to srg list of 2020-04-22)

1. Administrative 
   1. Presentation to EB
   2. Discussion w/CODATA
2. Reviewer spreadsheet/recruitment
3. Dagstuhl planning (in light of the pandemic)
4. Webinars for DDI-CDI review

MRT meeting minutes 2020-04-22

Attendees

Achim, Arofan, Dan G., Flavio, Hilde, Jay, Larry, Oliver, Wendy

Apologies

Jon

1. Administrative 

a. Presentation to EB

Arofan will report about the DDI-CDI release at the next EB meeting on May 4th.

b. Discussion w/CODATA

Achim and Arofan are in the process of discussing recruitment of reviewers with CODATA. The MRT group sees this as a great opportunity for promoting the DDI-CDI.

CODATA has put information about the DDI-CDI Public Review Release on their website https://codata.org/.

Dan G. proposed to look into possibilities for collaboration with other bodies as well and will look into options.

2. Reviewer spreadsheet/recruitment

Achim has provided a set of spreadsheets for organizing the review work.

• An email-list spreadsheet, containing email addresses to send the release announcement to.
• A feedback spreadsheet capturing received feedback from possible reviewers.
• A reviewers by topic spreadsheet where group members can fill in proposals for people and organizations to contact for the review.

3. Dagstuhl planning (in light of the pandemic)

Achim, Arofan, Steve and Simon H. are planning two Dagstuhl workshops scheduled to October 2020. The goal of the first workshop is technical testing of the DDI-CDI. The MRT group and external experts and experts from different domains will be invited. The second is a follow-up of last year’s workshop on interoperability of metadata standards in cross-domain science.

The plan for DDI-CDI is to organise the public review first, and then to update the public review version based on comments from the review. An intermediate release is scheduled to August. The aim oft he first Dagstuhl workshop would then be to have an extensive review of the interim version.

In light oft he pandemic it is still not sure if the Dagstuhl workshops can be organized as face-to-face sprint meetings as planned, and virtual alternatives needs to be considered. A question is how the review groups then best could be formed.

Flavio shared his experiences from the GSIM review, where sprint like groups were convened and worked remotely for 6-8 hours a day for a 4 day’s meeting. According to Flavio this worked, but was less efficient than a face-to-face sprint meeting would have been. Jay also reported similar experiences from meetings he has been involved in.

Jay asked if a project related to the RDA-COVID 19 he is involved with could be used as an example at the first or the second workshop. When Jay explained his example further,  Arofan recommended the examples to be used in both workshops.

4. Webinars for DDI-CDI review

Webinars by topic/audiences will be organized to introduce interested reviewers to the DDI-CDI. Adapted slide decks will need to be developed. People from the MRT group were asked by Arofan to present at the webinars, as well as about experiences from using different webinar platforms. Achim and Arofan will follow up on this.

Training material developed and used for the webinars could usefully be forwarded to the Training group. Dan G. will be the contact for this. A question is whether related material should be provided to the training group before or after the DDI-CDI has been published.

MRT meeting minutes 2020-04-15

Agenda (as of email to srg list of 2020-04-15)

1. TC review finalization:

- Text for website (review)

- Text for DDI Community e-mail/list announcement (review)

- Text for non-DDI Community e-mail/list announcement (review)

2- Planning for reviewer recruitment -  DDI-CDI reviewers spreadsheet

3- Planning for Webinars

Attendees

Achim, Arofan, Dan G., Flavio, Hilde, Jay, Larry, Oliver, Wendy

Apologies

Jon

1. TC review finalization:

- Text for website (review)

Arofan, Wendy and Jared have been corresponding about the website text. A new page for developing products of the DDI Alliance has been created, where the text related to the DDI-CDI will be found. The draft text was discussed at the meeting, and two minor suggestions for amendments were agreed. The revised text was sent by Arofan to Jared post-meeting, and the web site was updated accordingly.  

- Text for DDI Community e-mail/list announcement (review)

Wendy had drafted a text for this, where the only thing to change was the end date for the review period, that was changed from the 30^th of June to the 31^st July. Wendy will send the text to Barry and Jared for sending to the DDI Community e-mail lists.

- Text for non-DDI Community e-mail/list announcement (review)

Achim has drafted an email for non-community groups, based on Wendy’s proposal. This will be sent to the groups by MRT, and will also be sent to Jared and Barry for their information.

2. Planning for reviewer recruitment -  DDI-CDI reviewers spreadsheet

The spreadsheet (on google drive) has been updated since the last meeting. Potential reviewers are grouped by topics. The plan is to organize webinars for the identified topic groups. See below.

3- Planning for Webinars

Arofan suggested the following regarding webinars for potential reviewers from the different audiences:

• Webinars to start one to two weeks after the public review release has happened.
• A simple tool could be used to conduct the webinars.
• The webinars should be organized for smaller groups of potential reviewers, content being adapted to each group.
• 2-3 MRT members are expected to take part in each webinar.
• Content should consist of a 20 minutes introductory DDI-CDI presentation, plus discussions.
• Name and email addresses of interested reviewers to be collected.

This was agreed by the MRT group.

Flavio and Hilde were asked to distribute DDI-CDI recently held, to the group.

Virtual MRT meeting 2020-04-08

Agenda (as of email from Achim to srg list of 2020-04-08)

• DDI-CDI: decision on ready to roll, announcement
• DDI Alliance website: see proposal in Arofan’s email from yesterday
• DDI-CDI reviewers spreadsheet

Attendees

Achim, Hilde, Jay, Larry, Oliver, Wendy

 Apologies

Arofan, Flavio, Jon

MRT meeting minutes 2020-04-08

Achim was moderating as Arofan had a time conflict.

DDI-CDI: decision on ready to roll, announcement

The DDI-CDI public review package is ready to be published, the comments from Wendy and the TC taken on board.

Wendy har prepared a general announcement email for the DDI Alliance lists and IASSIST. A question is whether the same email will be sent to other groups as well or if amendments would be needed. Achim will talk to Arofan about this.

DDI Alliance website: see proposal in Arofan’s email from yesterday

A text proposal for a new section on the DDI Alliance web site provided by Arofan 2020-04-07 was discussed. Larry had some minor suggestions for additions. Larry wrote back to Arofan post-meeting, and after input from the marketing group and the TC the text can now be regarded as final.

DDI-CDI reviewers spreadsheet

A new reviewer’s spreadsheet by topic has been developed by Achim with input from Arofan and Hilde.  Potential reviewers to address was discussed. Group members were asked to add ideas for reviewers to the spreadsheet (in google drive).

So far the following topics are listed:

• Official Statistics
• RDA
• W3C Semantic Web
• UML Modeling
• DDI Community
• Semantic Interoperability and
  Conceptual Framework
• Policy Monitoring Indicators
• Infectious Diseases
• Resilient Cities

The plan is to arrange webinars for each topic, or combination of topics. Each topic group will get an MRT contact person for the review.

Achim asked the group about useful tools for webinar usage Larry then explained about PowerPoint audio narration that he uses when teaching classes online, and about Keltura that can be used to make videos.

Zoom will be used for the webinars, unless a better suited platform will be available.

Other

NADDI will this year be organized as a virtual event, and Wendy informed that a call for items to the program will be sent out. Options for possible virtual DDI-CDI presentations for NADDI was discussed. Arofan had proposed that an introductory DDI-CDI tutorial could be useful as well.  

Virtual MRT meeting 2020-04-01

Agenda (as of email to srg list of 2020-04-01)

1. TC review package corrections/progress
2. Planning for reviewer recruitment - Hilde's spreadsheet
3. DDI Alliance Website text and presentation

Attendees

Achim, Arofan, Dan G., Flavio, Hilde, Jay, Larry, Oliver, Wendy

Apologies

Jon

MRT meeting minutes 2020-04-01

Note on scheduled Dagstuhl workshop, October 4 – 9

The DDI-CDI will be out for review soon. After the review period amendments based on outcomes from the review will be made, and a new intermediate version of the specification will be provided.

A Dagstuhl workshop with the topic ‘Review from multiple domains’ is scheduled to October 4 - 9. Achim and Arofan have started to plan for this regarding what to review and who to invite. In addition to people from different domains, the MRT group will be invited.

Some ideas for topics/perspectives are:

• Integration
• DataDescription
• Interplay with other specifications
• Computational application processes
• UML review/interplay with other domain agnostic specifications (W3C)

Arofan asked the MRT group to get back with ideas on topics and people to invite for the planned workshop.

1.      TC review package corrections/progress

Achim received a list of things to correct from Wendy. This has been fixed post-meeting and Achim has sent an update of the package to Wendy.

Topics were related to the following:

• Mix in diagrams in PDF and EA file.
• Verification of links in the package Overview.
• Diagram notation to be introduced before the diagrams.
• XML schema root. This should be CDI rather than DDI.
• Shorten pathnames for the XSD documentation. 
• Add a note on the review page (already existent in the package) regarding the Javascript issue when using the Filed Level documentation locally.
• Add a note related to the examples regarding possible limitations.
• Namespace: Achim has proposed to use URL for this as W3C clearly states that namespace should be an URI, while DDI-C and DDI-L uses a string for this. It was agreed to make the namespace an URL in line with the W3C recommendations, and to make a notification that this needs to be adapted in DDI-C and DDI-L too, so that all the products are in line with W3C.

2.      Planning for reviewer recruitment

At the last MRT meeting people were encouraged to come up with ideas for people/projects to contact regarding the review. Larry and Dan G. provided some input, and Achim sent a comprehensive list of projects and people to contact to the srg email list prior to the meeting.

It was agreed to arrange introductory webinars for possible reviewers/teams organised by topic.

Ideas for topic groups are:

1. Official Statistics
2. RDA
3. W3C/Semantic Web
4. UML Modeling
5. DDI Community
6. CODATA Semantic Interoperability and Conceptual Framework
7. CODATA Policy Monitoring Indicators
8. CODATA Infectious Disease
9. CODATA Resilient Cities
10. Others (to be assigned): CESSDA, SSHOC, DataOne, Machine Learning, Computational Replication

Achim, Arofan and Hilde will do follow up work on this.

3. DDI Alliance Website text and presentation

Arofan is looking into this.

Virtual MRT meeting 2020-03-25

Agenda (as of email to srg list of 2020-03-25)

1. TC issues with review package and managing their resolution
2. Planning for reviewer recruitment - Hilde
3. DDI Alliance Website text and presentation

 Attendees

Achim, Arofan, Dan G., Hilde, Jay, Larry, Oliver, Wendy

Apologies

Flavio, Jon

MRT meeting minutes 2020-03-25

1.      TC issues with review package and managing their resolution

Achim will be the gatekeeper of the package and will liaise with Wendy regarding this.

Wendy and the TC is looking out for information related things, specific XML things, obvious broken things and errors.

The review is scheduled to last till end of June. If there would be a delay of the release with a couple of days the period could according Wendy be expanded till the end of July.

One of the issues detected was related to the unzipping issues with long path names for the XSD documentation. Achim will provide a new package with shorter path names, and will add a related note on the review page. In the meantime, it works to unzip the package in a path with a short pathname like “c:\ddi”.

Wendy also commented that in the schema included in the first version of the delivery package an URL was used for the namespace, whereas xmlns for DDI products usually has the form ‘ddi:name_of_xsd:version number’. Achim and Larry suggested it would be useful to have something that could resolve directly for review purposes if a related web page is put up. Arofan suggested that common practice is used for now. Wendy will send an example of this to Achim.

The name of the xsd was then discussed. It was agreed to use CDI for this purpose (plus something to indicate that this is a review version?).

Another issue that has been detected pre-release regards differences between the diagrams in the PDF and the EA documentation. Achim will look into this as well.

Wendy will get back to Achim with further issues detected by the TC while this week. The release by end of March is still a goal.

2. Planning for reviewer recruitment – Hilde

Recruitment of reviewers for the DDI-CDI was discussed. MRT are encouraged to contribute ideas for possible reviewers.

Virtual MRT meeting 2020-03-18

Agenda

Finalizing the review package and looking forward to getting reviewers.

Virtual MRT meeting minutes 2020-03-18

Attendees

Achim, Arofan, Dan G., Hilde, Jay, Larry, Oliver, Wendy

Apologies

Flavio

Finalizing the review package

Arofan and Achim are preparing the package for download and online access. Achim will prepare a bitbucket store for online access and will liaise with Oliver regarding that. This task has been completed post meeting.

Wendy and the TC will need at least a week to prepare the package for the review. End of March is still a still a goal for the DDI-CDI to be out for the public release.

Looking forward to getting reviewers

A plan for approaching interested reviewers and teams is under development.

Virtual MRT meeting 2020-03-11

Agenda (as of email to srg list) of 2020-03-11)

1. Finalizing the review package revision and preparing for handling comments.
2. List of known issues/dispositions
3. FAQ and DDI Alliance Site
4. Planning for recruiting external reviewers

MRT minutes 2020-03-11

Attendees

Achim, Arofan, Dan G., Hilde, Jay, Larry, Wendy

Apologies

Flavio, Jon, Oliver

1.      Finalizing the review package revision and preparing for handling comments.

Arofan, Achim and Hilde had discussed the release preparations in advance of the meeting.

Packaging of deliverables: Arofan is currently working on the packaging, and will send the final package to Wendy accompanied by a list of changes.

Known issues: The package includes a list of known major issues. These will also be available as Jira issues under the DDI-CDI project. The issues listed relates to handling of identification, annotation, as well as the simplification of strings. It was agreed at the meeting to include these major issues as part of the public review deliverable for information purposes.

XSD and examples: The XSD has been updated by Jay to support examples. All examples now validate.

Diagrams in the HTML documentation: Achim will include diagrams from the EA file in the ralted UML HTML documentation.

Architecture document: Sequence of content could usefully be changed. This will however not happen until the intermediate release scheduled to July.

Credits: Arofan will prepare the credits document for the deliverable.

BitBucket storage: Achim will make the zip file available directly from BitBucket, and will provide a link to the repository.

Follow-up of reviewers: A discussion followed regarding follow-up of reviewers. Hilde will be the main contact for reviewers entering comments or issues by email, and will write back to them to let them know that that their comments have been received.

Wendy recommended that it would be useful to email commenters by email in where comments are entered as issues in Jira. This was agreed at the meeting. Hilde will follow up on this.

Further follow-up on comments and issues will depend on perceived needs related to specific issues.

Wendy proposed that no DDI-CDI issues in Jira are closed before the end of the review period. This was agreed.

2.      List of known issues/dispositions

A list of open MRT related issues exist in Jira under the DMT project. These will be handled after the publication of the deliverable for the public review release. Hilde will send a link to these issues to the group for comments. A small group will then review the issues and the related comments and close solved or outdated issues.

3.      FAQ and DDI Alliance Site

The web-site group will be contacted regarding this. Possibly also the marketing group.

4. Planning for recruiting external reviewers

Options for recruiting potential reviewers were discussed. Plans will be developed in the coming time.

Virtual MRT Meeting 2020-03-04

Agenda (as of email to srg list as of 2020-03-04)

1. Finalizing the review package and preparing for handling comments. (TC has identified a couple of issues, and there is a short list of known ones.)
2. Organizing for the future:
   1. Timeline (as discussed)
   2. Plans for engaging reviewers

MRT meeting minutes 2020-03-04

Attendees

Achim, Arofan, Dan G., Hilde, Jay, Larry, Oliver, Wendy

Apologies

Flavio, Jon

1.      Finalizing the review package and preparing for handling comments. (TC has identified a couple of issues, and there is a short list of known ones.)

Delivery package:

The public review delivery package for the DDI-CDI has been sent to the TC for internal review and publication.

Wendy has provided a set of comments where some have already been dealt with, and the some will be dealt with prior to the publication. She has also provided a draft of the review page for discussion (see discussion below), as well as a DDI-CDI review issues tracker in Jira.

The DDI-CDI delivery package will be made available from the review page when agreed updates have been made to the package and the page.

At the meeting Dan G. commented that it would be a good idea to include diagrams in the UML field level documentation to be made available. Diagrams could be included in a separate folder and made available as HTML picture element, for example. Arofan will look into this.

Arofan presented a to-do list of things that remains to be looked into before the package can be published. The list was discussed with Achim and Hilde in advance.

• Credits and licensing must be improved with wording. Arofan will take on this. Achim has sent the necessary information.
• Achim will update the EA file with EA diagrams used in the specification documents.
• Achim is working on a list of open issues that he will deliver to Arofan for editing. The list will make up part of the delivery package.

• Achim is providing a bitbucket repository for storage and retrieval.
• Improved Canonical XMI files need to be integrated in the package. They are already available in Goggle drive.

• Improved documentation sets for UML and XSD is updated and available in Bitbucket and links from there are provided. The online documentation will be made available from a HTML release note page included in the delivery package.

• Arofan will write a note on UML documentation that there could be Javascript security issues with local use.

• Jay will fix diagrams on Architecture document – Arofan will include them in the specification document.
• Arofan will send XML examples plus schema updates to Achim

Jay mentioned at the meeting that the schema location needs to be changed in his examples. A (model related) tweak on the XSD would also needed in order to make some of the examples work. Jay, Achim and Arofan will meet to address the implementation of the change.

 Review page:

Arofan, Achim and Hilde had reviewed Wendy’s proposal for the page in advance, and provided a proposal for changes to the draft page. The proposal included a.o. incorporating  information about known open issues in the delivery package, as well to provide a sub-page for providing review comments in Jira or by email. The sub-page to be accessible from the main page.  It was also proposed that links to the online filed level documentation are included in the delivery package itself. The proposal for changes to the draft review page was agreed at the meeting.

It was also agreed that the full delivery package will be provided from the page, and that the content Overview of the delivery package will be duplicated on the review page.

In addition Larry proposed to link up specification documents (Introduction, Architecture, Detailed Model, Examples) directly on the page. This was agreed as well.

Wendy will make the edits to the review page with input from Hilde and the group.

Other:

Achim will create a Google Group email address for capturing review comments sent by email. Hilde agreed to coordinate review feedback entered by email.

Hilde will liaise with Achim regarding the Google Group email address and the review follow-up, and send the email address to Wendy for publication on the review page.

2. Organizing for the future:

a. Timeline (as discussed)

According to Wendy the TC will take on the DDI-CDI publication for review while the week of March 16^th. This gives the MRT group a week to perform the final tweaks. Wendy stresses that the importance of having a clear approach for reviewers regarding what to do and how to go about to do it.

The next step will be to prepare for an interim July release where important comments from the review are taken on board. A working area will be set up in Confluence where a copy of the public review package will be worked on.

Virtual MRT meeting 2020-02-26

Agenda as of email to srg list as of 2020-02-06

1. Administrative items
2. Packaging Proposal for Review
   1. - Outline (see attached)
   2. - Credits and licensing
3. Review page
4. Open issues list

MRT meeting minutes 2020-02-16

Attendees

Achim, Arofan, Dan G., Flavio, Hilde, Jay, Larry, Oliver

Apologies

Jon, Wendy

1.      Administrative items

Deliverables for the release: Arofan is happy with the material for the deliverable received from the group. Things go well together and the specification looks more coherent than earlier DDI specs. The final editing of the spec. for the public review release is currently ongoing.

Dagstuhl workshops: Achim informed there will be two DDI related Dagstuhl workshops this autumn.

• The first is a DDI-CDI workshop, focusing on DDI-CDI review from multiple domains. This workshop will take place from October 4^th through 9^th.
• The second workshop will have focus on interoperability of metadata standards in cross-domain science, health and social science application. This workshop is organized in collaboration with CODATA.

The MRT group will be invited to take part in the first workshop. Attendance for the second workshop is still under discussion.

2. Packaging Proposal for Review

a. - Outline (see attached)

Arofan sent out an outline of the DDI-CDI Download Package by email to the srg group on February 26^th. The content of this email is included in the Appendix.

It was agreed in the group to distribute the complete delivery package in a single download. This will be made accessible from a HTML landing web page that will be referenced from the TC review site. An overview of the deliverables explaining the content of the package will also be made available from the landing page. Arofan will be writing the text for this.

b. - Credits and licensing

Credits: The following was agreed:

• To list MRT members and their institutions.
• To list all DDI Moving Forward, DDI4 and MRT Sprint workshops since 2012 and the organizations contributing to their funding.
• To list all previous (DDI Moving Forward/DDI4) working groups.
• To mention Dagsuhl, that hosted nine DDI4 related workshops, subsidized costs and provided free seminar rooms.

Achim will draft a proposal for the crediting and send to Arofam.

Licensing: The DDI-CDI will be licensed under aCreative Commons Attribution 4.0 International License.

Achim has provided Arofan with further details.

3. Review page

Wendy is developing a draft of the TC review page in collaboration with Arofan and Hilde.

Wendy has also suggested to provide a CDI project in Jira for the purpose of capturing the comments from the review filed as issues. This idea of Wendy’s was agreed in the group. Hilde will ask Wendy to set up the CDI issue tracker in Jira.

Arofan and Hilde will further get back to Wendy regarding the TC review page when the work on the delivery package is completed. Hilde and Wendy will then coordinate the publication and send out info to the group.

Jay raised the question whether we will know who downloads the package for the review. There was a discussion regarding this, and an agreement was made that it would be better to be able to download the package anonymously, without having to provide an email address.

On the other hand it would be good to know the email addresses of people providing review comments in Jira. Hilde will ask Wendy if this will be possible.

4. Open issues list

Achim is working on an open issues list that will be provided to Arofan on Friday 28^th.

Other

Normative/non conformative references: Which references are normative and which are not has not been discussed so far. Dan G. was asked to do a job on this for the final release, starting after this release. This is particularly important for the sections of the DDI-CDI specification that regards Standards alignment and the Foundational part.

Upper model: The Upper model will not be made available in the Canonical XMI and for consistency purposes not in EA either. It will however be described in a related chapter of the specification document. Related diagrams will be added to the document.

Appendix

Email from Arofan to the srg list as of 2020-02-26:

Outline of the DDI-CDI Download Package

Directory: DDI-CDI Public Review 1

• File: Overview.htm describing and linking (locally) to package contents
• Directory: Specification Documents
  • Files: Document I – IV of the specification, in PDF format
  • Directory: Supporting Documents
    • UML Subset document
    • Others as needed
• Directory: Model
  • Files: 2 XMI versions, one with unique relationship names, one without
  • File: Enterprise Architect file
  • File: Release notes (if needed) in PDF
  • Directory: Field-Level Doc
    • Files: HTML Field-Level doc for the model
• Directory: Syntax Representation
  • File: XSD
  • File: Release notes (if needed) in PDF
  • Directory: XML Schema Doc
    • Files: HTML XML Schema documentation
  • Directory: Examples
    • Files: XML examples
• Directory: Examples (if needed)
  • File: 00ReadMe.txt
  • Files: Any examples other than the XML examples (which would be referenced from the ReadMe.txt file)

Virtual MRT meeting 2020-02-19

Agenda as of email to srg list of 2020-02-19

1. Administrative items
2. Documentation status

- Field level

- High level

1. Model freeze/final modifications
2. Process for updating diagrams, round 2

Attendees

Achim, Arofan, Flavio, Hilde, Jay, Larry, Oliver, Wendy

Apologies

Dan G., Jon

MRT meeting minutes 2020-02-19

1.      Administrative items

Product name

The new product name DDI Cross Domain Integration, or DDI-CDI has now been approved by the EB.

Barry has asked Arofan for an update of the on the DDI product page and the related FAQ to be provided.

The working name DDI4 Core will be changed to DDI Cross Domain Integration/DDI-CDI in all deliverables for the public review release.

Review site preparation

Wendy is preparing a standard TC public review site as a placeholder for the public review delivery package. She will liaise with Hilde regarding this. A link to the site was provided post meeting. A list of deliverables will be provided.

2. Documentation status

- Field level

Hilde has been updating the field level documentation of the ClassLibrary of the frozen EA file (of 2020-02-16), based on input from MRT colleagues. Detected issues will be flagged as known in a related release document.

Post meeting notes: The documented EA file version available was on Friday 21. Based on the file Achim has produced and made available related deliverables (Canonical XMI, XSD, EA documentation etc.).

- High level

Roughly all needed documentation sections are provided, but need editing. As the release date is rapidly approaching a thorough pre-release MRT review will not be possible timewise.

Achim is working on a document explaining some of the aspects of UML, for example what the arrows mean.

The specification will have a form closer to that of other specifications (e.g. W3C) compared to earlier DDI releases.

2.      Model freeze/final modifications

The frozen model need some final modifications before publication for the public review release that regards which packages to keep in the file. It was agreed at the meeting to remove the ModelManagement section and the SupportingDiagrams from the public review version of the file. Regarding diagrams, see 4. below.

RDF will not be available for the public review release, but Eric Prud’hommeaux will be contacted regarding this after the release.

4. Process for updating diagrams, round 2

Larry has created diagrams for the DataDescription package from the frozen EA file. The diagrams are in EMF (vector) format. They will be integrated in the frozen EA model if possible.

A discussion about the correspondence between the diagrams in the deliverable documents and those of the EA model took place. It was agreed that a correspondence between the diagrams in the model and those of the documents is needed to avoid confusion. Larry suggested that identical diagrams in the EA file and the documents carry the same name, which was agreed.  

Jay pointed out that currently different color codes are used for diagrams from different sections. Color codes will need to be standardized in the future. Larry proposed that for now each document section could include information about the color codes used as a reusable footnote. This approach was agreed by the group.

The format of pictures was also discussed. EMF vector files are useful, in particular when larger diagrams are displayed. A question that arose is whether there would be a MAC problem related to EMF or if relevant documentation formats to be used supports EMF. This needs clarification and Larry will liaise with Arofan regarding this.

Achim will look into the diagram sections of the frozen model with the aim to rationalize and get the diagrams better organized. It was agreed that all current and used diagrams should live in a single DDI-CDI Library diagrams folder in the EA file.

Arofan, Achim and Jay will liaise further regarding diagrams for the public review release.

Other

There was a discussion about outreach, and ideas that came up were to offer webinars/conference calls for interested parties, and as a post public review release activity, to contact organizations and groups that have shown interest in this new product of the DDI Alliance.

Virtual MRT meeting 2020-02-12

Agenda (as of email to srg list of 2020-02-12)

1. Administrative items
2. Documentation status

- Field level

- High level

3. Model freeze/XML representation

4. Process for updating diagrams

MRT meeting minutes 2020-02-12

Attendees

Achim, Arofan, Dan G., Hilde, Jay, Larry, Oliver, Wendy

Apologies

Flavio, Jon

1.      Administrative items

Product name

Arofan has sent a mini-report to Barry in the Marketing team that reflects the naming discussion in the MRT group. A new name for the DDI4 Core is needed in all the deliverables for the public review release, therefore a swift decision is required.

2. Documentation status

- Field level

Hilde has now received revised rsd files from most of packages for the release from her colleagues. Feedback on Long (Flavio) and StructuredDataTypes (Oliver) is expected in the near future.

When the frozen version of the Master EA file from Achim (scheduled to Friday 14^th evening) is available she will start the field level documentation upgrade on the file.

- High level

Data Description document

Hilde and Arofan have been working to reorganise and update the Dagstuhl version of the document. The document needs further refreshment of diagrams and their related content.

How to read the model?

A new document chapter about how to read the model will be developed. This will include explanations of what the arrows mean, as well as other content aiming at facilitating the understanding of how to read the model for new users.

Foundational document

This is fine for now. It will be revised when the other things are in place in order to make it consistent with the rest.

Architecture document

Achim will provide more content for this. Flavio is working on the CollectionPattern. Jay has provided the standards alignment part of the document, but will check this for updates.

Use-cases

Use-cases will be an ongoing activity. Jay has started this and 3 – 4 examples are now available in XML.

Dan G.’s Time Series example will also be included – including XML if time allows, otherwise it will be published as a conceptual use-case.

The examples will be made public when a storage has been provided for them.

2.      Model freeze/XML representation

Master EA model and examples

Jay and Achim has tested the model and the examples. Now things work!

The tested version includes Process updates from Jay as well as agreed model clean-ups from Achim. Main changes to Process regards the Scripting representation. The changes allow Process to do a lot of things, for example to handle processes like that of C2Metadata.

Process now integrates nicely with DataDescription and can act on it’s classes, as well as on the more foundational classes like variables, codelists etc.

The InformationObject class from the UpperModel will now be moved to DataDescription (Achim).

The Upper Model

The UpperModel is not part of the deliverable other than as an explanatory diagram. This needs to be verified to make sure the connection to the ongoing work is kept. After the public review release the upper model needs to be documented.

Model freeze

Achim will provide an EA Master model version for freeze on the evening of Friday 14^th. Some minor clean-up changes will be implemented before the freeze.

3.      Process for updating diagrams

Diagram presentations and updates

Achim explained about a feature in the EA that allows export of all diagrams in various formats including .png and .jpg. Larry suggested vector formats could be used. It was agreed to use .png format for the public review release and consider vector format later.

Larry pointed out that not all diagrams in the documents stems from real EA diagrams. Jay’s diagrams are snapshots of the model copied to clipboard to make them available in each section of the document. They will, however, be turned into EA diagrams after the public review release.

EA diagram updates

Larry agreed to update the diagrams for the Data Description document. The procedure will be as follows:

• Larry receives the frozen version of the Master EA that will be sent out on the evening of Friday 14^th. This will have been updated with a folder named ‘Data Description diagrams’ for entering of the updated diagrams.
• Larry makes a copy of the file where the suffix ‘has diagrams’ is added to the file name.
• He deletes outdated diagrams available in the ‘LH’ environment of the Sandbox.
• Updated diagrams for Data Description (with a specific name describing what is in it) provided by Larry will be stored in the ‘Data Description diagrams’ folder.
• Achim will include the updated DataDescription diagrams in the frozen Master EA file if possible to merge in. This would happen (if possible) after the documentation upgrade has been done.

Larry suggested that it would be good to have a connection between the diagrams in the EA model and the representation of the diagrams in the documents. Achim suggested that each diagram is given a specific name describing what is in it, and that the same name is used in the Word document. This was agreed by the group.

The work of the MRT group for the public review release continues in good progress.

Virtual MRT meeting 2020-02-06

Agenda

Follow-up from meeting 2020-02-05

MRT meeting minutes 2020-02-06

Attendees

Achim, Arofan, Hilde, Jay, Larry, Wendy

Apologies

Dan G., Flavio, Jon, Oliver

Arofan opened the meeting by saying that what would be needed for the prototype release is a model and a schema that people can understand and work and play around with. Lots of good work has been done on DataDescription, Process and the VariableCascade and it is important to get this out to people. Syntax representations can come later on. The Review process will bring useful input to the final product. What the model (PIM or PSM) is called is not important at this stage. In addition to the deliverable a test space could be provided. The model and the schema needs to be frozen for the release now.

Latest modeling updates: Achim received an updated version of the model that included needed changes for the Process model part that facilitates handling of components from DataDescription and Foundational. Achim has produced a Canonical XMI and an XSD (not valid) on basis of this. The Process model needs some further upgrade. In this take identifiable information will be added in on relevant classes, in order to be able to make examples. Jay will fix things that led to the invalid XMI and perform further changes and will liaise with Achim on Monday 10^th regarding the model. A new version of the model including updates to Process will be available on Tuesday 11^th . Testing can continue in a separate sandbox.

Model clean-up: When Achim receives the file from Jay on Tuesday he can continue with the model clean-up as far as time permits. On Friday 14^th he will hand the file over to Hilde who will start to edit the field level documentation (until Saturday 22^nd).

The following agreement was reached regarding what to edit, based on Achim’s email to the srg list as of 2020-02-05:

1. Leaf classes without attributes and associations to other classes
1. ControlStep, ScriptingAgent and RulesEngine - Drop
2. Sequence and Service - Leave
3. Designation - Drop
4. What is the purpose of UnitSegmentLayout? – Leave
5. PhysicalSegmentLocation has no attributes, only one association to ValueMapping, and only one sub class, SegmentByText. Currently it doesn’t make much sense. Should this be maintained for future expansion? – Drop for now
2. The current approach of string types seems to be too complicated. The structured string types could be folded into the regular ones. Furthermore, there are some inconsistencies.
   Diagrams on the current approach and the suggested approach are attached. – Merge existing string types if time allows.
3. Missing identifiers for classes in DataDescription, Process, and value? of a Code , see Jay’s email on “new model version 2020-02-04 / generated files” from 2020-02-04.
   Attached is a list of classes in DataDescription and Process. The classes which inherit from Identifable are highlighted. – Jay will add identifiers to DataDescription and Process. - Achim will liaise with Flavio on whether something needs to be changed in the other packages.
4. Identification and Annotation. Identification could be an attribute not a super class (otherwise “, annotation could be a separate class. – Implement identification as super classes for now and change to attributes later on. - No change regarding handling of Annotations.

Branding: A name for the product will be needed for next week. Arofan will follow up on this and send the MRT naming discussion to Barry.

 Virtual MRT meeting 2020-02-05

Agenda (as of email to srg list 2020-02-05)

1. Administrative items

- Branding

- FAQ/Site

1. Field-level doc status
2. High-level doc status
3. Model fixes/freeze (including XML Representation)

Attendees

Achim (last part of the meeting), Arofan, Dan G., Flavio, Hilde, Jay, Larry, Oliver, Wendy

Apologies

Dan G., Jon

MRT minutes 2020-02-05

1.      Administrative items

- Branding

   Arofan will send the MRT email discussion thread on branding to Barry. Then the Marketing group will then decide for a name on the consent of the EB.

- FAQ/Site

   Barry in the Marketing group has also asked the MRT group to update the Work Products of the DDI Alliance page with new content on the DDI4 Core (which name will change before the public review release), as well as the integrated FAQ. This task will be taken on after the     public review release has happened.

2.      Field-level doc status

Hilde has received Agent and Identification package rst files from Wendy, Process from Jay, Conceptual, Representations and FormatDescription from Dan G., as well as Components, Dimensional and the root classes of DataDescription (Datum etc.) from Flavio.

Flavio has also agreed to do Wide and Key value. Oliver agreed to look into StructuredDataTypes.

Hilde would need the remaining updated content packages in hand by the 12^th be able to integrate the content in the field-level documentation for the deliverable.

3.      High-level doc status

DataDescription part: Arofan and Hilde are looking into this.

Process: Jay is responsible for this.

Foundational: The document is good but needs some positioning. Arofan will contact Dan G. regarding this.

Architecture: Achim and Jay have provided content for this. Flavio’s chapter on design patterns to follow. A draft needs to be available by February 22nd.

4.      Model fixes/freeze (including XML Representation)

In advance of the meeting Achim had sent a proposal for alternatives for model fixes with attachments by email to the srg list (see the Appendix). This includes among others a proposal to treat Identifiable as attributes rather than super-classes. A decision is needed regarding what of this to implement now, and how.

Jay argued that when testing Process issues were revealed that would require more testing to make this part of the model workable, and wanted some timespace for this.

Arofan pointed out that the model needs to be frozen on the 14^th for the field level documentation to be done, and that Jay needs to liaise with Achim regarding possible changes to the model.

Arofan proposed that a Wiki page will be made available where examples can be updated along with the review, this way separating the normative from the non-normative parts of the deliverable. This was agreed.

A discussion about how to handle identifiers followed. Options for how to fix inheritance for now discussed at the meeting were:

• Add identification as attributes for classes that miss identification, and change identification by inheritance to identification as attributes for those classes where this applies.
• Add identification as attributes for classes that miss identification and leave classes with identification by inheritance as they are.
• As a temporary solution, add identification by inheritance to classes that miss identification (Larry).

Larry argued annotations should be handled needs a further discussion. Arofan then proposed to decouple annotation from identification.

Jay expressed no strong opinions of this, while Arofan argued for the value of Achim’s proposal to make identifiable attributes to avoid class hierarchies in the model. Simple IDs could be implemented for all instances.

The approach was discussed in relation to time. Arofan argued Achim had indicated willingness to do the job and that this should be discussed further with Achim on his arrival at the meeting.

Jay argued then that Process would needs some further modeling and wanted space while the coming week for testing and brainstorming with Flavio till next Wednesday.

This would leave Achim only two days for the model clean-up fixes. On his arrival at the meeting Achim indicated that this would not leave him enough time to implement identification consistently and do other fixes.

Time did not allow the group to reach agreement on whether Process modeling can continue before the release, how to deal with identification, or on other of  Achim’s proposals for clean ups.

Therefore, a follow-up meeting on Thursday 6^th at 11 Ottawa & New Hampshire/17 CET was scheduled.

Appendix

Email from Achim to the srg list 2020-02-05:

Here is a list of open model issues.

The goal would be that we either find today an agreement or we describe the issue and possible solutions and park the issue.

I’ll then make changes to the model according to the agreed decisions.

1. Leaf classes without attributes and associations to other classes
   1. ControlStep, ScriptingAgent and RulesEngine
      1. Jay: they can be removed
   2. Sequence and Service
      1. Jay: need to remain as is in line with idea that this is a conceptual/logical model that some but not all will use for implementation too (Smile).
         Sequence is a container for other ControlConstructs and because it inherits from ControlLogic which is reflexive maybe Sequence can host other ControlLogic and itself.
         Service is an idea we would like to finish at another time.
   3. Designation
      1. Ja: Designation is another touchpoint with Signifier, Sign and Signified
      2. Flavio: We are not using Designation anymore

• Dan: Designation isn’t really needed since we have the signification pattern. As a result, each application of designation, whether it be a code, term, or other, will be modeled in the realization. Designation, being a generic term, probably is not needed.

1. What is the purpose of UnitSegmentLayout? Does it make sense for future expansion.

1. PhysicalSegmentLocation has no attributes, only one association to ValueMapping, and only one sub class, SegmentByText. Currently it doesn’t make much sense. Should this be maintained for future expansion?
2. The current approach of string types seems to be too complicated. The structured string types could be folded into the regular ones. Furthermore, there are some inconsistencies.
   Diagrams on the current approach and the suggested approach are attached.
3. Missing identifiers for classes in DataDescription, Process, and value? of a Code , see Jay’s email on “new model version 2020-02-04 / generated files” from 2020-02-04.
   Attached is a list of classes in DataDescription and Process. The classes which inherit from Identifable are highlighted.
4. Identification and Annotation. Identification could be an attribute not a super class (otherwise “, annotation could be a separate class.
   The current model uses Identifiable and AnnotatedIdentifiable as super classes. The Identifable is just “Data-Based Inheritance”. There are currently 16 classes which inherit directly from Identifiable and 23 classes which inherit directly from AnnotatedIdentifiable. There are overall 110 classes which inherit from other classes and finally from Identifiable.
   The information which is captured by Identifiable can be modeled as a data type which could be used as attribute with the name “identifier” for each class. Even the information which is captured by AnnotatedIdentifiable can be modeled as a data type. The exception is the association to AgentAssociation. There are four m:n associations from AnnotatedIdentifiable to AgentAssociation, each with some kind of role (publisher, contributor, versioning agent, creator). The weird thing is that AgentAssociation itself has again a role item (external CV).
   I see here some requirement for discussion how this could be simplified. Diagrams on the current approach and one suggested approach are attached.
   1. The information of Identifiable and AnnotatedIdentifiable could be collapsed into one structured data type with the exception of AgentAssociation.
      Annotation could be an attribute of Identifier. This would be more limited than the current solution in terms of the annotation relationship.
   2. Another approach would be to have associations from classes which need Annotation. One annotation could be then used for many classes. A related super class could be constructed. The latter approach would make distinction between identification (which would be an attribute) and annotation (to which an association would exist).
   3. A third approach would be to include the identification information in the super class described in b). This is similar to the current approach but Annotation is not tied to a single class. This seems to be a flexible solution.

Achim

Virtual MRT meeting 2019-01-29

Agenda (as of email to the srg list as of 2019.01.29)

1. Administrative items
2. Tasks for review finalization
3. Field documentation status
4. Document status
5. Model clean-up (Achim's proposal)

MRT meeting minutes 2019-01-29

Attendees

Achim, Arofan, Dan G., Flavio, Hilde, Jay, Larry, Oliver, Wendy

Apologies

Jon

1.      Administrative items

Branding: Arofan and Wendy met with the Marketing Team that proposes to have a new name for the DDI4 Core. Arofan will get back regarding what is decided. (Lively discussions by group members regarding the name followed post meeting).

2.      Tasks for review finalization

The delivery date for the TC is set to February 28^th. Arofan will need to have the frozen model with field level documentation included on the 19^th by the very latest.

Achim will have the navigation cleanup done by the end of the week and will provide an updated EA file, Canonical XMI, XSD and rst files.

When Hilde receives updates field level documentation files (rst) from the group members she will include them in the updated rst files. Achim will continue with model clean-ups till this is done and content is ready to be copied into EA.

For documents, see the documents section.

3.      Field documentation status

Hilde had received updated Agent and Identification packages from Wendy, Process package from Jay before meeting. Dan delivered updated Conceptual, Representations and FormatDescription files  post-meeting . Input from Flavio on the DataDescription packages and StructuredDataTypes is expected in the coming days.

4.      Document status

DataDescription: This needs restructuring and an introduction. Work is expected to happen in the coming days. Hilde and Arofan will liaise on this.

Foundational: A cross-walk from the UpperModel to this is needed. The document needs to get a little more easily accessible. Dan G. will look into this and finalise within the next two weeks.

Architecture documents: Jay has responsibility for the standards alignment. Achim takes care of the UML sub-set used as well as PIM/PSM. Flavio on the CollectionsPatterns piece.

Examples: Jay and Chifundo are currently working on examples on data management (HDDS), standards alignment, and data transformations between formats. Some of the examples are dependent on a valid XSD, and some possible iterations may be required.

Flavio mentioned that a ‘fake object model’ also could be useful, to have something on the model level.

It remains to be decided which of the examples to include in the model.

Documents final for review by the MRT group will be made available on the new Public Review Preparation pages in Confluence.

5. Model clean-up (Achim's proposal)

Achim has a proposal for further model clean-ups, among those the role of Identifiable and AnnotatedIdentifiable as super classes. Identification Instead could be made an attribute. And how to handle Annotation best? Regarding Annotation Larry commented that it is important that several object can have the same annotation. Achim will lay out his proposal by email to the group later.

Virtual MRT meeting 2020-01-22

Agenda (as of email to srg list of 2020-01-22)

1. Administrative items/feedback from EB
2. Status of the specification/overview of current docs
3. Plan for finalizing the model and completing inline doc review/incorporation
4. Plan for finalizing specification docs and deliverables
5. RDF syntax representation discussion

MRT meeting minutes 2020-01-22

Attendees

Achim, Arofan, Dan G., Hilde, Jay, Larry, Oliver, Wendy

Apologies

Flavio, Jon

1.      Administrative items/feedback from EB

EB meeting: The report provided by Arofan for the EB meeting on January 21^st was well received.

Achim informed there was a naming discussion and it was decided that the Marketing group will liaise with Arofan to find a new name for the DDI4 Core.

Timeline: The public review deliverable will be sent to the TC on February 28^th for release. The work on content of the MRT group should be final by the 14^th. The TC will need some time for the publication after the review package has been received.

A longer period is expected for the public review. Wendy informed that four weeks is the minimum requirement of the TC for a review period, while three months is an average.

The MRT group is working on a timeline to prepare for the final public review release. The timeline will be made public at a later stage.

2.      Status of the specification/overview of current docs

The documents sent to the EB need upgrade for the release.

The introduction document is now quite good, but group members are still welcome to feedback comments or proposal for amendments.

The other documents need further amendments – see point 4 below.

3.      Plan for finalizing the model and completing inline doc review/incorporation

An open issue under discussion is whether to have a merely conceptual PIM would impact on how to handle directionality in EA.

For the public release, however, a functional and tested XSD with a corresponding variant of the model would be needed.

Arofan proposed the following approach for the public review:

• The latest version of the model in EA (as of 2019-12-19) will be used as a starting point. A variant of this will be created that implements the changes needed to produce a Canonical XMI that will serve as basis for a workable XSD. The changes to the current version of the EA file includes fixing the directionality of aggregations and compositions, as well as other changes that may be needed to be able to produce a workable XSD representation.

  Achim was assigned the task to change the directionality of aggregations and compositions as well as to liaise with Oliver and Jay regarding other possible changes. Iterative XSD testing and other checks may bring up issues that needs to be fixed in the model.

  A discussion on whether the amended variant of the model will be called a PIM or a PSM will depend on the changes made and will be taken up again later on, providing pros and cons of each.

  Diagrams based on the 2019-12-19 version of the file could possibly be provided that documents how a pure conceptual PIM would look like. Then people can input on what they would like to have while the review. This needs a decision.

• The documentation upgrade of class, property and attribute level documentation, will be performed on the updated variant of the model, as it would not be enough time to implement upgrade the documentation in two different variants of the model for the public release.

  Hilde will enter the revised documentation based on input as provided by the group members (rst files by package). So far rst files for Identification, Agent and Process has been provided. How to handle Enumerations, RegularExpressions and XMLSchemaDataTypes       needs  to be looked into. Achim will provide some advice.

The proposal of Arofan above was accepted by the group. No one objected strenuously against this proposal.

4.      Plan for finalizing specification docs and deliverables

The documentation upgrade work with the rst continues. Flavio will be asked to look at StructuredDataTypes in addition to the packages already been assigned to him.

The following tasks regarding upgrade of documents were assigned at the meeting:

Foundational: Dan G. was asked to provide a text piece bridging from the upper level documentation to the detailed Foundational document created at Dagstuhl.

DataDescription: Hilde was asked to look into the DataDescription part and make suggestions for ordering of content and needed updates of diagrams and content to be compliant with the latest version of the model. Larry already started this work.

The transformation between data structures might be included as usecase example in the Appendix (suggestion by Arofan).

Process: Jay and Arofan will look into this. A data management usecase example would be useful to include if possible.

Standards alignment: Jay has some new content developed by Cifundo and himself that regards mapping of DDI4 Core to DDI Codebook/NESSTAR, as well as to schema.org. Jay will liaise with Arofan regarding inclusion of this content.

Release timeline:  A release timeline, including steps towards the final public review release, is under planning by the group. This will be made public made when final.

5.      RDF syntax representation discussion

Eric has come back to Achim regarding transfer of the model transfer to RDF. It looks like it would now be possible to produce an RDF OWL representation based on the Canonical XMI. Achim will come back with more next week.

Virtual MRT meeting 2020-01-15

Agenda (as of email to srg list as of 2020-01-15)

1. Administrative items (if any)
2. Versioning issue update
3. In-line documentation
4. Model status & updating (directionality, etc.)
5. Document deliverables: outline/intro update

MRT meeting minutes 2020-01-15

Attendees

Achim, Arofan, Dan G., Flavio, Hilde, Jay, Larry, Wendy

Apologies

Jon, Oliver

1.      Administrative items (if any)

Tuesday January 21^st is the deliverable deadline for the progress report for the EB. The deliverable will contain an introduction document with a list of deliverables, high level documents on the Core detailed model, architecture, detailed examples and use-cases, XMI and XSD files, as well as field level documentation.

Achim will provide XMI, XSD, documentation and rst files for the EB deliverable. He will tentatively use the December 19^th version of the model for this. If immediate agreement is reached regarding how to handle navigability in the UML spec, and the decision involves to change the directionality of aggregations and compositions manually in EA, Achim has offered to take a stab on this if time allows, based on a frozen version of the model.  

The delivery of the review package will come later. We’ll need to have the work on the content for the package ready by mid-February. Then some days will be needed for the packaging and sending to the TC. The review package needs to be done with by end of February. Together with the deliverable a set of known issues will be published.

2.      Versioning issue update

The latest published version of the master model is that of December 19^th. Flavio will check if further amendments have happened since then.

Flavio explained that ‘Manage versions’ in Google Disk has been used to manage versions of the Master EA file. See how in the appendix.

Arofan asked Wendy for TC recommendations on where to handle version control of delivered files for the public review. Wendy recommended to use Bitbucket for this purpose.

3.      In-line documentation

Work is ongoing and Wendy has provided proposals for the Agents and Identification packages. Dan G. will work on FormatDescription, Conceptual and Representations.

The rst files provided for the work by Achim is based on a version of the model available by December 15^th. On some of the packages, for example Process have undergone important changes since then and the forthcoming updated rst files for those would helpful to complete the work.

Hilde is coordinating the class, attribute and association level documentation. She has also been asked by Arofan to do the copy/paste of revised documentation into the model in EA for the review version. The class, attribute and association level documentation will be further reviewed during the public review process of the deliverable.

Arofan pointed out that the UpperModel is meant as an introductory diagram for the review package. Class level documentation is thus not important for this purpose. However, at some point Jay provided some class level documentation for the UpperModel, which is not available in the current version of the master model. Jay and Flavio will liaise regarding this at a later point in time after the review package delivery.

4. Model status & updating (directionality, etc.)

An issue relating to navigability and directionality in the model that needs to be solved prior to the review deliverable was discussed at the meeting. Achim offered to change direction of aggregations and compositions manually in EA if there would be agreement in the group for that. Larry agreed to write up a proposal for how the navigability issue could be solved. This was sent by email to the srg list after the meeting. Achim responded positively to Larry’s proposal, while follow-up from Flavio is expected in the near future.

5.      Document deliverables: outline/intro update

Prior to the meeting, Arofan sent an updated version of the introduction document for the EB report to the group for comments, which includes main updates from Jay on the Upper model and Process.

In an email to the group prior to the meeting, Achim proposed some changes to the first introductory pages of the document, which in his view should convey the main messages to a broader audience.

Key features could be described as:

• domain-independent
• flexible data description (based on the description of a single datum/cell, also for new data type)
• provenance/process description
• foundational metadata (Variable Cascade, Format Description, Classifications)
• UML for interoperability (other standards) and sustainability (other representations)

These and other ideas of Achim's were approved in the group, and Arofan will take a stab to update the introduction document accordingly.

Appendix

Versions of the Master file on Google Disk can be found by clicking on the newest version of the file, and then on the ‘timer’ on the column to the right. See figure below.

Virtual MRT meeting 2020-01-08

Agenda (as of email to srg list of 2020-01-08)

1. Administrative items (if any)
2. Status and timeline
3. In-line documentation
4. Model status & updating
5. Document deliverables: outline/intro

        (1) The exact outline of the delivered spec (which is described in this document)

(2) Marketing and branding - there has been some discussion of this. We need to make a decision for the review at least. Also, I wanted to come up with a slogan for the header.

(3) Process of updating the diagrams. These (and consequently their explanation) is, I think, out of data. What will the process for finalization be?

(4) How do we credit members of the group? What licensing text do we use?

MRT meeting minutes 2020-01-08

Attendees

Achim, Arofan, Dan G., Flavio, Hilde, Jay, Larry, Oliver, Wendy

Apologies

Jon

1. Administrative items (if any)

There will be a meeting in the EB on January 21^st. Jared has asked for an update by then on where we are with the review deliverable. The update will be provided by Arofan in due time.

The next step is to finalise the review deliverable. Hilde agreed to prepare a new milestone page for the work in Confluence.

2. Status and timeline

Everybody is now back from vacation.  21^st of January will be too early for finalising the review delivery package, but work is gradually approaching this goal. February is a rough estimate for the deliverable.

3. In-line documentation

Work to manually upgrade the Class, Association and Properties documentation has started. Procedures are described in DMT-318. reStructuredText files for each model package provided by Achim on December 15th is used as basis for the work, as described in issue DMT-318. Possible issues affecting the current work is that the model has changed since the rst were produced (latest version 2019-12-19), and that some content from Lion might not have been transferred successfully to the current model.

Achim, Flavio and Hilde will look into how to best follow up on this.

Achim proposed additionally to separate definitions in reStructuredText from examples, synonyms, mappings etc. and provide a hierarchy of documentation files for different purposes and audiences. This idea is important and needs follow-up.

4. Model status & updating

The latest version of the Master eap file for the DDI4 Core was provided by Flavio on January 19^th. Decisions need to be made regarding directionality in EA, that needs to be set in order to get the navigability for aggregations and compositions right in the export to XMI and XSD. This is specific to EA.

In his emails to the srg list of 2019-12-17 Achim proposed to change the directionality of aggregations and compositions manually in the eap version of the PIM. Flavio agrees to do the manual update of the directionality in EA, but would prefer to make the model variant with set directions a PSM. Achim, however, would prefer not to make the PIM dependent on specific EA solutions. Achim and Flavio will look at this together, based on Achim’s findings on the topic from December and two examples from Flavio, and get back with a recommendation to the group.

It also needs to be decided whether Identifiable and AnnotatedIdentifiable sould be defined as structured data types and used as properties in the related classes, as suggested in issue DMT-298 . Achim and Flavio will look into this as well.

5. Document deliverables: outline/intro

(1) The exact outline of the delivered spec (which is described in this document)

Arofan sent out a revised version of the DDI4 Core Model introduction document to the group on the meeting day, that contains a proposal for an outline of the review version of the specification.

The group was asked to go through the outline till next time and provide feedback.

(2) Marketing and branding - there has been some discussion of this. We need to make a decision for the review at least. Also, I wanted to come up with a slogan for the header.

Comments have been raised regarding use of numbers in the names of DDI versions (DDI2.5, DDI 3.2, DDI4 and so on). To find a new product title for the ‘DDI4 Core’ needs to be considered.

(3) Process of updating the diagrams. These (and consequently their explanation) is, I think, out of data. What will the process for finalization be?

Some changes have been made to the model since Dagstuhl that affects the diagrams provided and the related documents. Arofan will get back to the group regarding contribution to the upgrade of diagrams and related documentation.

(4) How do we credit members of the group? What licensing text do we use?

Wendy sent an email to the group before the meeting where she reports about an existing agreement for licencing under CC BY 4 for all DDI products. In the email she also refers to the XKOS and DDI Lifecycle Readme as two examples of acknowledgement of contributors.

At the meeting the group was asked by Arofan to provide ideas on how to best acknowledge contributors for the DDI4 Core review deliverable.

Virtual MRT meeting 2019-12-18

Agenda (as of email to srg list of 2019-12-18)

(1) Timelines and deliverables reporting to EB

(2) Modelling issues and impact on XML representation

(3) Documentation clean-up

MRT meeting minutes 2019-12-18

Attendees

Arofan, Dan G., Flavio, Hilde, Larry, Wendy

Apologies

Achim, Jay, Oliver

(1) Timelines and deliverables reporting to EB

Reporting is due January 21^st. Arofan will report about the state of the art of the DDI4 Core in due time.

(2) Modelling issues and impact on XML representation

Flavio has been making good progress on the model clean-up. The new CollectionPattern is now in place and DataTypes have also been looked into. See email from Flavio to srg list of 2019-12-19 for a description of the post-meeting (2019-12-19 ) version of the EA Master file.

Navigability in aggregations and compositions when transformed to XMI is an open issue. Achim suggested in email to srg list of 2019-12-17 a way to handle this that consists in manually changing the ‘Direction’ in EA before export. At the meeting Flavio proposed an alternative approach that would be to programmatically change all aggregations and compositions in EA to associations. In a post-meeting email to the srg list of 2019-12-19 Flavio however expresses support of Achim’s approach.

To have the navigability fixed is important in order to provide XSD files for testing by Jay and possibly others. 

Another open issue regards the handling of Identifiable and AnnotatedIdentifiable that currently adds 1-2 additional levels in the inheritance chain. They could be defined as structured data types and used as properties in the related classes, as suggested in issue DMT-298. It has been agreed earlier to implement this.

Decisions on approaches to handle the navigability and inheritance related issues will be made when everybody is back from vacation.

(3) Documentation clean-up

The documentation per class, attribute and association needs review and update. The work needs to be carried out externally to the EA, for manually inclusion in the EA Master file later on.

In his email to the srg list of 2019-12-15 Achim provided documentation templates per class to be used for the documentation clean-up. Each template contains one package. The template indicates where necessary editing of the documentation per class, attribute, or association should be made. Edited and agreed sections will later be updated manually by copy/paste to the EA model.

The syntax of the templates is restructuredText (like in the prototype documentation). Achim suggests to use the online reStructuredText editor tool http://rst.ninjs.org/ for this work. This was agreed at the meeting.

Hilde agreed to coordinate the documentation per class, attribute and association update work, and will contact MRT members individually to assign tasks. It was agreed at the meeting that each package will be updated by one MRT person and reviewed by a different one. Alternative proposals for changes in the class, attribute and association documentation will be discussed in the group when needed. Procedures for the work is further specified in issue DMT-318 (created post meeting). The work is expected to start after the vacation.

Virtual MRT meeting 2019-12-11

Agenda (as of email to srg list of 2019-12-10)

1. EDDI report
2. Modelling issues from the list/model status
3. XML Representation testing
4. Open issues from list

Attendees

Achim, Arofan, Dan G., Flavio, Jay, Larry, Hilde, Oliver

Apologies

Jon, Wendy

1. EDDI report (Achim)

DDI4 at EDDI 2019

Achim chaired the session on the DDI4 Core at EDDI.

Flavio presented remotely in the session that went very well. In his presentation Flavio gave a high-level overview of the DDI4 Core, and provided examples of potential use of the DDI4 Core at StatCan (see Flavio's abstract).

After Flavio’s presentation Hilde presented data description with the DDI4 with examples. Her presentation was written with Larry (see Hilde and Larry's abstract).

For both presentations we got positive feedback from the audience. Our impression was that simple slides work best, and that new architectonical things take time to present. For a presentation of 20 minutes, 20 to max 25 slides would be appropriate. Further supplementary material can usefully be added to the end of the presentation for questions and later reference. The topics of the two presentations work well together but for a next time an idea could be to integrate them tighter. Another question is how to best present the capabilities of the DDI4 Core.

In addition to chairing the DDI4 Core session, Achim held a tutorial on generating syntax representations from the DDI4 Core with the Eclipse tools (see Achim's abstract). The slide deck for the tutorial contains more than 100 slides, and got positive feedback from the participants. Achim will get back to the group later regarding ideas related to this.

Achim also held a 10 minutes presentation of the DDI4 Core at Conference Plenary of EDDI.

IASSIST 2020

For IASSIST abstracts for similar presentations to those held at EDDI have been submitted by Flavio and Hilde. An idea is to include an example where process is used to convert e.g. wide format to aggregate in the data description presentation for IASSIST.

In addition Dan G. has submitted a proposal to present usage of the DDI4 Core data description in the modernization processes of the BLS.

2.      Modelling issues from the list/model status

Flavio updated the master model DDI4CorePIM EA file before the meeting 2019-12-11. Updates include:

• Work on the new CollectionPattern, moving all realizations into the new one. 2-3 classes remain to be updated.
• The DataStructures package has been renamed to DataDescription.
• Along with comments by Flavio in Jira issue DMT-315, LogicalDataDescription have been moved to the Legacy folder, some used classes like Datum and DataPoint has been moved to the DataDescription package root. LogicalRecord (abstract) is now a collection of InstanceVariables. This has now been moved to FormatsDescriptions.

By the meeting on Wednesday 11^th, cleanup of data types still remains to do on the model (see issue DMT-301). Wendy (not present at the meeting) offered earlier to check which are used based on XMI.

This could be done on the basis of the latest Canonical XMI provided by Achim.

Flavio will continue to work on the CollectionPattern and the StructuredDataTypes in the coming days. He will also fix some errors detected by Achim (see of email to srg list 2012-12-11 or the Appendix).

The ProcessPattern also needs to change location and be moved to a package as this is no longer a pattern. Larry commented that we need to make sure that abstracts are tagged.

To do

Flavio will continue to correct errors and provide two variants of the master PIM model before the weekend, one with Process moved to the ClassLibrary as a package and one without.

It should also be looked into how UML can be adjusted to be able to map more directly to schema.

3.      XML Representation testing

Achim provided an update of the XMI prior to the meeting, as well as an XSD file. The is not valid due to detected errors (see the  Appendix).

An important issue regards usage of non-unique association names.

When Flavio has provided an updated EA file (see 2 above), Achim will create a new Canonical XMI file with unique association names on the level of the model on the form subjectName-predicateName-objectName, possibly using machine generated acronyms for subjectName and objectName.

Achim, in collaboration Oliver, will then develop a new version of the XSD file based on the updated Canonical XMI. Non-XML data types needs to be fixed on the level of the schema.

When available Jay can then use this for test purposes.

4.      Open issues from list

Documentation structure

In email to the srg list of 2019-12-04, Dan G. provided a proposal regarding which documentation to include on the level of the class, property and association (see Appendix).

Some further discussion regarding this will be needed, as class name for example equals ID in Canonical XMI.

To do  

An XSLT transform of Canonical XMI content to markdown, applying the (possibly amended) proposal regarding which documentation to include on the level of the class, property and association from Dan G., could be used as a structure to add and edit content.

Edited content needs to be flagged.

Achim will look into this, after the Canonical XMI/XDS work is done. Discussions with Dan G., Larry and possibly others will be needed to complete this work.

Appendix

XSD issues detected by Achim; Extract of email to srg list 2012-12-11

Below are a couple of issues (visible in the XSD) which should be fixed in the model.

Achim

textContent (of StructuredDataTypes::DynamicText) is defined as DynamicTextContent

name (of DataDescription::Dimensional::DimensionalDataSet) is defined as objectName

name (of DataDescription::Dimensional::DimensionalDataStructure) is defined as ObjectName

overview (of DataDescription::Dimensional::Revision) is defined as InternationalStructuredString – maybe in wrong package defined?

frequency in DataDescription::Dimensional::ScopedMeasure is defined as char

name in DataDescription::Dimensional::ScopedMeasure is defined as ObjectName

changed to xs:string in XSD as workaround

data types in XMLSchemaDatatypes are not recognized as xs data types

type="anyURIType" changed to type="xs:anyURI"

type="languageType" changed to type="xs:language"

should be fixed in the transform to XSD

has (and others) are ambiguous as association names

Approaches for fixes must be discussed.

Email Dan G. to srg list 2012-12-04

MRT,

Here is a follow up to the email I sent last week about documenting the model. I propose the following template:

Class

ID

Name

Definition

Properties

                Name

Definition

Cardinality

Optionality

Datatype

Relationships

                Pointer to Relationship table

Example

Relationship

ID

Source Class ID

Cardinality

Optionality

Target Class ID

Cardinality

Optionality

Semantic

Name

Definition

Relationship type (relation: association, subtype, composition, aggregation)

Maybe we can consider this via email before the next meeting.

By optionality I mean whether something is required, optional, or conditional.

Dan

Virtual MRT meeting 2019-11-27

Agenda (as of email to srg list of 2019-11-27)

1. Administration (EDDI, etc.)
2. Status of drafts
3. Status of model
4. Status of code deliverables (XMI, XSD, etc.)

MRT meeting minutes 2019-11-27

Attendees

Achim, Arofan, Dan G., Flavio, Hilde, Jay, Larry, Oliver, Wendy

Apologies
Jon

1. Administration (EDDI, etc.)

EDDI 2019: Achim will chair the DDI4 session at EDDI.

Flavio will give the presentation ‘DDI 4 Core: describing and managing data for traditional and modern data platforms’ remotely. Hilde will subsequently present ‘Data description with the DDI 4 Core‘.

Achim will additionally give the tutorial titled ‘Generating DDI 4 in the Language of your Choice’at EDDI.

The presentations and the tutorial can later be used as supplementary material to the DDI4 Core deliverable.

IASSIST 2020: The deadline for abstracts for IASSIST is December 6th. Possible contributions from the MRT group were discussed. Dan G. has already sent in a proposal.

Flavio agreed to submit an abstract based on the same topic as that of the EDDI presentation.  Hilde agreed, depending on travel possibilities, to send in a proposal for a tutorial on data description in the DDI4 Core (hands on approach with actual instances).

2. Status of drafts

Arofan continues to work on the chapters for the DDI4 Core specification.

Achim will start to work on the architecture document after EDDI. He will also update and complete the part of the modeling guidelines that was created in Ottawa.  

Furthermore the examples need to be looked at.

3. Status of Model

Model update: Flavio will start the modeling upgrade work  again this week. The first things he will look into is the new collection pattern and the data types (see issue DMT-301).

Wendy offered to help to check where they are used. Flavio will send Wendy a check list of what he needs in advance.

Class/properties/association documentation:

A discussion about documentation on the class/properties/association level resulted in the following agreements:

Agreements

• Definitions, examples and explanatory notes will be added for classes.
• Synonyms, mappings to DDI 3.2, GSIM and RDF that was available for classes in Lion will be removed to a separate document. The ‘Trace’ relationships in the model will be used to identify mappings between DDI4 Core classes and classes belonging to other standards.
• For associations and properties only descriptions are needed.
• Larry will look into possibilities for generating UTF text files as structured as markup for all content (one folder pr. package). Then documentation on the class/properties/association level will be easy to review, see what is missing etc. Larry will use the same software as used for the DDI4 R package. If the work of Larry turns out to be complicated Achim and Oliver will look into other possibilities.
• New and updated content could be added in the text file and should be flagged with an asterisk, for example. New/revised content can then be entered in EA. Jay agreed to take on that.
• Dan G. together with Larry and Hilde will provide guidelines for forming definitions.

4. Status of code deliverables (XMI, XSD, etc.)

Flavio will let Achim know when the next version of the model has been published. Achim will then create a new Canonical XMI and send to Oliver. Oliver will then produce a new XSD file.

If the update of Flavio takes time Achim and Oliver will make the updates based on the current version of the master model (of November 17).

Sphinx will continue to be used for production purposes. Oliver is almost finished to bring Sphinx to use Canonical XMI.

Later Jay will start to test  the model on real examples.

Other

Correspondence between the DDI4 Core and DDI-C on the variable level

In Dagstuhl correspondence with DDI-C on the level of the variable was discussed, and it was agreed to add a defined set of classes with descriptive text to the model for the January release. These classes will be deprecated and removesd  along with further DDI4 developments.

How to relate to DDI-C on the study level is a bigger discussion that will happen post-release.

Meetings

The next meeting is December 11th. This will be the last MRT meeting of this year.

Virtual MRT meeting 2019-11-20

Agenda (as of email to srg list of 2019-11-19)

1. Administrative items
2. Progress on the model (Flavio)
3. XMI/Syntax representation issues
4. Update on issues (Hilde)
5. Upper model discussion (Jay)

Attendees

Achim, Arofan, Dan G., Flavio, Hilde, Jay, Larry, Wendy

Apologies

Jon, Oliver

MRT meeting minutes 2019-11-20

1. Administrative items

After this autumn’s Dagstuhl workshops the DDI4 Core is on its way to become an additional product of interest to several organisations and projects. One example is the ALPHA 2.

A related product, the DDI4 libraries R, has also become a product of interest.

The increasing interest from different parties makes it even more important that the MRT group delivers the specification within a due timeframe.

2. Progress on the model (Flavio)

Flavio has updated the ‘DDI4 Core-Master.eap’ file (last update 17th November).

He reports that the structural changes have been implemented. He has also be working on bug fixing and clean up related to the latest issues, as well as on updates on diagrams.

The class/association level documentation remains to be looked into.

Data types: In issue DMT-301 point 4. Achim reports that some classes are indeed only data types (no associations).They should be defined as structured data types not as classes.

We know which data types are wrong, but not where they have been used. An XMI scan can help to sort this out.

Documentation:

Classes/associations: Dan G. looked into definitions for  the Conceptual, Representations and LogicalDataDescription packages at the Dagstuhl workshop.  There are still parts of the model that lack content. In addition everything has to be reviewed.

Word or a spreadsheet can be used for updates and content can then be then copied to EA.

HTML documentation from EA will serve as basis to find out where documentation is lacking as well as for the review.

Defintion development rules need to be decided,  and a template for class/association level documentation upgrade and review needs to be provided.

The field level documentation in Lion containes more documentation on the level of the class than just the definition, for example practical examples. A similar template could be used for the DDI4 Core class/association level documentation. The template can be retreived  from markdown from Lion.

Agreements

• Data types; Issue DMT-301: Flavio will write to Achim and specify what he needs for the XMI scan. Achim will then provide Flavio with lists of things that needs to be fixed based on the scan. Wendy also offered to help with this. Flavio will make the necessary amendments based on the feedback from Achim.
• DMT-303, DMT-315: Flavio will provide lists for what needs to be checked.
• The next step for Flavio will then be to finalise the implementation of the new collection pattern. This is likely to happen in December.
• Dan G. and Hilde agreed to look into class/association level documentation development rules and to provide a template for the documentation upgrade and review
• Jay has offered to provide a document about the new collection pattern documentation.
• Achim agreed earlier to write the model architecture piece of the spec.

4. Update on issues (Hilde)

The below issues were discussed. Comments made at the meeting for each issue are found in the table of the appendix. It was decided to set DMT-302 and DMT-300 (both in green in the table below) to ‘resolved’.DMT-300: We need a clear description how aggregation is used in contrast to regular associations.

Virtual MRT meeting 2019-11-13

Agenda as of email to srg list as of 2019-11-13.

1. Administrative items
2. Proposed outline for specification (Arofan sent today)
3. List of open issues (Hilde)

Minutes from MRT meeting 2019-11-13

Attendees:

Achim, Arofan, Jay, Hilde, Larry, Oliver

Apologies:

Dan G., Flavio, Jon, Wendy

1. Administrative items

Model clean up process: Flavio is continuing the clean-up work on the Master model with good progress. He will publish a new version of the Master file on Friday. In the next couple of weeks he will have less time. Larry has offered to help with this work.

EDDI presentation: There will be a meeting in the EDDI 2019 Programme Comittee on the forthcoming Monday. We expect that our proposal for Flavio to present remotely, with assistance from the session chair for the QA session, then will be discussed and hopefully accepted.

2. Proposed outline for specification (Arofan sent today)

Prior to the meeting Arofan sent out a proposal for an outline of the DDI4 Core specification (see Appendix 1). The proposal is based on the fact that the DDI4 Core specification is model specific rather than schema specific.

Two variants of the model will be produced, one that includes the design patterns and one without patterns. Two variants of the Canonical XMI will be produced, one with unique association names within packages and one with non-unique association names.

The proposed outline consits of a series of documents/files (1) through (5)  where each are separate documents/downloads:

(1) The Core Model

(2) Architecture and Use

(3) Detailed Model Documentation

(4) XML Schemas

(5) RDF/OWL Vocabulary (Prototype)

Agreements

-It was agreed to organise content in folders. Each top-level  section will have a top-level folder that contains sub-folders with content.

-Based on Achim’s proposal it was agreed to split up (2) Architecture and Use in two as these are very different topics.

-Proposed sequence for (1) Core Model sub- folders:

                Introduction

                Upper model

                Data Description

                Foundational

                Process

- Proposed sequence for (new 2)  Architecture:

                - Architecture

                - Patterns

                - External standards

                - Other DDI Specifications

- Detailed documentation on the Class and association level  will only be made available in clickable HTML. Achim will look into options for this.

-Doc-Flex or Oxygen will be used for XML based files. Achim will look into the options.

-A guidence document needs to be provided for the January release.

 -Modeling rules will be included in an appendix. Arofan reminded the group that Eric Prud’hommeaux has expressed interest in the sub-set of UML that the DDI4 is using. This is something Achim will look into after the production of the specification.

-Arofan will revise the outline and send the revised proposal to the group and will take the lead to compile the spec.

3. List of open issues (Hilde)

Hilde provided an overview of Jira issues created since Ottawa (DMT-273, DMT-281 – DMT-288, DMT-294-DMT-303). See comments from the discussion in Appendix 2.

Many of the issues regards detailed changes to the model and we need to discuss with Flavio how to deal with them.

Appendix 1

Outline for Organizing the DDI 4 Core Specification by Arofan as of 2019-11-13

I propose a series of documents/files - (1) through (5) are each separate documents/downloads:

(1) The Core Model

1. Introduction
2. Upper Model
3. Data Description
4. Process
5. Foundational Metadata

(2) Architecture and Use

1. Examples
2. External Standards
3. Other DDI Specifications
4. Patterns in the Model
5. Architecture

(3) Detailed Model Documentation

                [HTML Clickable documentation]

(4) XML Schemas

(5) RDF/OWL Vocabulary (Prototype)

Appendix 2

Virtual MRT meeting 2019-11-06

Agenda (as of email to srg list as of 2019-11-05)

1. Update on the DDI 4 Core Summary document
2. Modelling issues discussed at the MRT meeting 2019-10-30, issues updates from Achim
3. Other

MRT meeting minutes 2019-11-06

Attendees

Achim, Arofan, Flavio, Jay, Hilde, Larry, Oliver, Wendy

Apologies

Dan G., Jon

1. Update on the DDI 4 Core Summary document

Arofan has sent the ‘Looking forward to DDI4 Core’ summary document to Jared for forwarding to the EB and Barry in the Marketing group.

The document will be revised for an external public later on.

2. Modelling issues discussed at the MRT meeting 2019-10-30, issues updates from Achim

XSD schema:

Oliver has provided a more or less validating schema base don the latest Canonical XMI from Achim.

Detected issues related to the XML schema are recorded by Achim in issue DMT-303 ‘Model issues in transform to XSD: multiple defined properties and associations in super class/concrete class, missing data type etc.’.

According to Oliver these issues do not require much work to fix.

Oliver asked if document information is required to describe the context of the XML instance. It was agreed in the group to wait with this till January, and feed back to Oliver when this is ready for inclusion.

Model issues checks by Achim:

Achim performed several checks on the model since the last meeting . The following was discussed at this meeting.

Overall impression: No major issues

The checks reveal several minor things that need to be cleaned up, but no severe errors in the model were detected. Flavio will start on the tidy up work and make some structural changes to the current file regarding Design Patterns and packaging.

Association names

It was agreed to stick with the agreement of last MRTmeeting  to provide two variants of PIM, one with unique- and one with non-unique association names.

Quoting from Achim’s comment at issue DMT-299:

MRT meeting of Oct 30, 2019: same association names in one package are allowed for the purpose of readability.

There will be a related general purpose version of the Canonical XMI.

A second Canonical XMI version could have unique association names based on the existing association name combined with the related class names. This version can be consumed by UML tools which don’t allow non-unique association names in one package.

The agreement made at MRT meeting of Oct 30, 2019 was  confirmed at this meeting.

Unique association names should have the form ‘subject class name + association name + object class name.

Achim will try to implement programmatically Hilde’s idea to use acronyms for the subject- and object class names. This has second priority, though.

Navigability of aggregations and compositions

At the MRT meeting of Oct 30, 2019 it was tentatively agreed to remove navigability from associations in the model. Based on analyses by Achim regarding how unnavigability of aggregations and compositions perform in different UML tools and in the export to XMI, the group decided at this meeting to keep the navigability of aggregations and associations, but to create excisting aggregations and compositions anew.

We currently use the navigable end (arrow) to indicate the direction of associations. In the case of associations or composition navigability should be on the level of the ‘part’, that is at the opposite end of the ‘whole’ (indicated by the diamond). The opposite end is usually ‘unspecified’ in the current model. Some UML tools can have issues with that. Achim reports that in EA there is sometimes no distinction to see between unspecified and navigable or between unspecified and navigable. In Magicdraw unspecified navigability in an imported XMI results in a non-navigable state,which is an issue.

According to Achim a clear way to avoid the problem would be to first create a regular association in EA with an arrow on the ‘part’ side and then set the aggregate side to ‘shared’ using theassociation menue (role). The aggregation end should be set to aggregsation=shared.

There are currently 21 aggregations and 34 compositions in the model.

It was agreed that Flavio creates the aggregations and compositions anew in EA to get things standardised and correct.

PIM/PSM distinction

Moving towards January the role of the PSMs in relation ot the PIM will be more pronounced. Actual implementations will make the distinction between PIMs and PSMs clearer. Achim reports that a document regarding the PIM/PSM difference is in the process. This document will make up an important part of the specification.

3. Other

Flavio’s  EDDI presentation:

Flavio has submitted an abstract for a DDI4 Core related  presentation at EDDI that has been accepted by the programme committee .

Unfortunately it turns out that it will not be possible for Flavio to attend the EDDI conference in person this year . Flavio and the MRT group mean that the best alternative would be that Flavio gives the presentation remotely, with assistance from the session chair, and with backup possibilities.

Achim agreed to convey the view of the MRT group to the EDDI Programme Committee.

Dagstuhl week 1 delivery package for the EB:

Arofan has been preparing the delivery package for the EB from the DDI4 Core Sprint at Dagstuhl to be made available for the EB for their upcoming meeting.

MRT meeting 2019-10-30

Agenda (as of email to srg list of 2019-10-29) :

1. Administrativia
2. DDI 4 Core overview draft
3. Status on specification documentation
4. Status on the model
5. Status on syntax representations 
6. Modeling guidelines - planning for changes

Meeting minutes virtual MRT meeting 2019-10-30

Attendees

Achim, Arofan, Dan G., Flavio, Hilde, Jay, Larry, Oliver, Wendy

Apologies

Jon

2. DDI 4 Core overview draft

Arofan has sent an extended revision of the DDI4 Core summary document for review by the group. The document is scoped for the DDI community. He is expecting comments through Friday November 1st. The document will be sent to the EB in good time for their next meeting on November 7th. The MRT group is author of the document. The names of the group members will be included in a footnote.

3. Status on specification documentation

Larry has added a document about ‘Streams’ to the Dagstuhl Wiki page in Confluence. Arofan will include this as an example for the DDI4 Core report from Dagstuhl. Experts in the field will be asked to review later.

4. Status on the model/5. Status on syntax representations 

Achim has provided  a Canonical XMI file for the group to look at.

Oliver will use this to produce XSD files while this week. As Views are no longer in the XMI he will need to adapt the transformation to the new structure.

Both the XMI file and the XSD are planned to be included in the Dagstuhl delivery package for the EB.

6. Modeling guidelines - planning for changes

In advance of the meeting Achim had posted three isues related to modeling rules:

DMT-298 Inheritance chain - transform Identifiable to property defined as structured data type.

DMT-299 Association names must be unique in one package. Association names should be mandatory.

DMT-300 Aggregations don't add semantics. They should be removed. Some compositions have issues.

DMT-299 and 300 were discussed at the meeting.

DMT-299 Association names

 In Canonical XMI identifiers are based on association names. If they are non-unique within packages this is a real issue. Quoting from DMT-299:

• Associations without names could be considered by some UML tools as associations with same names which is not allowed.
• Many syntax representations need a name for an association. It doesn't make sense that each transformation for these syntax representations invent names. This can be error prone if different names are created.

According the modeling rules from Ottawa: each association should be understood as triple where the association name is the predicate and the object is navigable (arrow) in UML. The triple should be read as English sentence. This supports the understanding of the model and syntax representations like OWL/RDF.

        Many association names don't fulfill this requirement.

The status of the current association names is provided in the spreadsheet ‘Associations.xlsx’ attached to the issue.

In advance of the meeting Achim made a proposal for to solve this (sent by email to the srg list prior to the meeting). Requirements and the proposal is quoted below:

       There are multiple (possibly competing) requirements regarding association names:

1. Readability of associations in diagrams and documentation, i.e. no long names with class names included
2. Support of reuse of associations in some syntax representations like OWL/RDF (i.e. same associations should be recognized programmatically while the transformation)
3. The identifiers in Canonical XMI must be unique. These identifiers have a semantic structure based on the names of items like associations.
4. UML doesn’t allow non-unique names of items (like associations) in the same package. Some UML tools complain about this, some ignore it.

       Proposal which tries to address all requirements

1. The requirement one can be understood as the guidance because this has first priority. There is already the rule that associations - with the connected classes – should be understood as a simple sentence with subject (class one), association (predicate), and object (class two, navigavable, i.e. line has an arrow). The used association names should be reviewed according to this.
2. Same association names should be allowed in one package if they have really the same meaning. They could be then reused in a syntax representation. The reuse could be across packages.
3. The unique XMI ids can be generated while the transformation to Canonical XMI according to the rule that the ids are structured like “class1+assocation+class2”.
4. A second Canonical XMI version could be generated with unique association names according to the structure mentioned above with the ids. Unique association end names (roles) could be here addressed in a similar manner.
   This version can be offered for consumption by UML tools where the other Canonical XMI version produces errors.

Agreements

The full proposal was agreed by the group. Consistency needs to be handeled on the     level of the PIM.

Default names for missing association names for aggregations and compositions  should be set to ’has’.   

Task

Flavio will add non-excisting association names (‘has’) in EA for aggregation and composition , and upgrade other association names to comply with English grammar rules. He will also remove references to class names in the association names where this occurs in the EA PIM. The spreadsheet  ‘Associations.xlsx’ in DMT-299 serves as basis for the work.    

The upgraded EA file will serve as basis for the XMI related implementations in 3. and 4. of the proposal.

General comment regarding consitency checks

The reason for naming aggregations and compositions is to ensure consistency across implementations of the PIM.

Flavio comments that it is important to note that the DDI 4 Core spec goes beyond the PIM and that similar issues involving potential implementation inconsistencies arise elsewhere, e.g. constraints that are not machine-actionable, extensions of UML datatypes, etc. These could be interpreted differently by different developers creating interoperability problems similar to those of unnamed associations. We need to look at all the places in which there are non-machine-actionable constructs and find ways to ensure implementation consistency.

DMT-300 Aggregation/composition

Aggregation

The discussion regarding aggregation started with that it is not clear in our model what is the difference between aggregation and multiplicity >1 at one end. Either the difference should be explained or aggregation should be dropped. For software this would be the same thing.

Flavio explained that aggregation is just for users who look at part/whole relationships. This is only a visual help and has no programmatic impact.

Jay pointed out that aggregation is part of the UML specification and can be used for this reason.

Achim suggested that the type of model should impact on whether aggregation is used or not. If the model is explorative modellers recommend to include aggregations. If the model is an implementation model they recommend not to use them. Flavio proposed to view the DDI4 Core PIM as explorative as it is a conceptual model, while the PSMs would be closer to implementation models. On this basis it was agreed to keep aggregations in the PIM.

Aggregation, composition and navigability

Larry raised a possible issue regarding navigability of aggregation and composition that there might be differences in the model between things imported from Lion and drawn in the EA. As normal XMI parsing of associations is based on navigability. Flavio suggested to drop navigability of aggregations and compositions. This was tentatively was agreed by the group at the meeting.

Larry was asked to follow up on this and provide Achim with examples. Post meeting Larry found out that there was no real issue with this in the model (quoting from Larry’s  post meeting emai to the srg list):

                The role properties for these associations are consistent between imports from Lion and aggregations or compositions drawn in EA. Only the “Source” and “Destination” are  swapped.  In both cases the Navigability property is set to “navigable” on the end opposite                the one where the Aggregation property is set to “composite” or “shared” (the diamond end).

Post meeting Achim wrote (in a post-meeting email to the srg list):

We use the navigability of associations in the implementation of representations for the purpose of identifying (and versioning) an object. What does this then mean in the case of a composition without navigability? We might want to establish a rule how the navigability should be realized in the representation for a composition, i.e. a composition has a composite (which is navigable).

Agreements

Based on the fact that some group members find aggregation useful and that the DDI4 Core model can be perceived as explorative it was decided at the meeting to keep aggregations in the model.

Tentatively agreed to remove navigability in aggregations and compositions (see also the comments from Larry and Achim post-meeting above).

Task

Flavio and Achim will work together to document the use of aggregations and composition in the model.

MRT meeting 2019-10-23

 Agenda (as of email to srg list 2019-10-22):

1. Update on Sprint documents/overview
2. Report from TC meeting (Jay/Larry/Dan) - ?
3. Status on Model
4. Status on specification documents
5. Use of GANTT charts (Hilde)

Meeting minutes virtual MRT meeting 2019-10-23

Attendees

Achim, Arofan, Flavio, Hilde, Jay, Larry

Apologies

Dan G., Jon, Oliver, Wendy

1. Update on Sprint documents/overview

Arofan will provide a summary report early the coming week at latest. This will represent a snapshot of where we are. He will include an intro to the DataDocumentation part. The Patterns document will be included when the model is stable.

2. Report from TC meeting (Jay/Larry/)

DDI variants: There is currently consensus in the TC to continue to have multiple DDI variants in the future.

DDI4 Core – DDI-Lifecycle : Wendy started to work on a high level mapping.

COGS: COGS in its current state would not yet be suitable for adaption by the group for model  based production and management puropses, as some current model content would not be supported. It would for example not be possible to impose restriction on ‘source’ cardinalities for associations.

3. Status on Model

Larry proposes that newly developed structural components are included in EA to facilitate the production of the Canonical XMI. This was agreed by the group. It was also agreed to implement the new package structure agreed at Dagstuhl.

Achim will produce Canonical XMI based on the next stable version of the model (which includes the new structural components and has implemented the agreed package structure from Dagstuhl). He will then talk to Oliver regarding the implementation of syntax representations.

Achim and Larry will do analyses on the current model version in Canonical XMI and provide a list of things that need to be updated for the January release (like critical associations, aggregations, compositions etc.) or classes that need to be moved back from the Legacy folder of the current version of the model . The list will be distributed to everybody in the group and discussed by Achim, Flavio and Larry in a separate meeting.  Agreed changes need to be implemented and followed up to check  that everything has been done.

Another issue regards the direction of diamonds in compositions and aggregations. When drawing diagrams in EA, EA assumes a direction. However, Achim explains, ‘source’ and ‘target’ does not excist in UML.  Arrows have navigability that should be based on the predicate as well as the  versioning direction. New developments and amendments to the model since the prototype need to be looked into to check out usages of compositions/aggregations in relation to navigability.

Flavio will look into the Modeling guidelines to check out agreements regarding usage (or not) of aggregation, and check out the needs to update the model and/or guidelines.

Achim proposes that more guidelines on how to use EA is needed. This was agreed by the group.

4. Status on specification documents

The introduction document and the DataDescriptopn pieces have been reviewed by different group members.

Larry will write a part on Conceptual Value for January.

Dan G. s Time Series document will be included as an example. A short version of the Foundational document will be provided  for January.

5. Use of GANTT charts (Hilde)

Hilde tried out a free Gantt project tool for follow-up on the MRT tasks in the Core delivery phase. She will look further into tasks reporting possibilities in Jira and get back to the group with options.

MTR meeting 2019-10-16

Agenda as of email to srg list 2019-10-15

1. Taking Lion offline?
2. DDI 4 Core - DDI Lifecycle mappings: planning for TC Minneapolis meeting
3.  Review of Sprint and status report (including 2nd Dagstuhl week presentation)
4. Plan for moving forward on deliverables

MTR meeting minutes 2019-10-16

Attendees

Achim, Arofan, Dan G., Flavio, Hilde, Jay, Larry, Wendy

Apologies

Oliver, Jon

This was the first MRT using Zoom, link https://zoom.us/j/641763395

1. Taking Lion offline?

It was agreed to keep Lion up for a couple of weeks still as this could be useful for variuos purposes (review, complex data types, referencing for example).

According to Wendy it is no problem to keep Lion up for some more time.

2. DDI 4 Core - DDI Lifecycle mappings: planning for TC Minneapolis meeting

Mapping to earlier versions of DDI will be topic at the forthcoming TC meeting in Minneapolis. According to Wendy mapping will then be looked into with a broad focus. The discussion could feed into the work of the MRT group. A goal would be to be able to map DataDescription and  Process/Provenance  of the DDI4 Core to earlier versions of DDI (DDI-Codebook; DDI-Lifecycle), and  to have mappings available for the January release.

3. Review of Sprint and status report (including 2nd Dagstuhl week presentation)

First Dagstuhl week

Arofan summed up shortly from the 1st Dagstuhl week . All  goals of the meeting agenda were achieved.  The RDF representation, classified as ‘Nice to have’ in the agenda, remains open.

Second Dagstuhl week

Achim reported about great interest of the DDI4 Core as an integration model at the second dagstuhl week. The work started at week 2 will continue, with the goal for the DDI4 Core to be used in a larger context.

Arofan, Larry and Dan G. explained  about synergies with CODATA work, interest from other research domains in how the DDI4 Core deals with data types,  concordances with ontologies etc.    Interest was also shown by representatives from schema.org.

4. Plan for moving forward on deliverables

Hilde sent the following documents by email to the srg list prior to the meeting:

1) Dagstuhl Sprint overview of deliverables and tasks.docx - This document contains an overview of published deliverable related content, provides a summary of forthcoming tasks and lists Jira issues filed since the Ottawa Sprint.

2) Dagstuhl discussion regarding DDI4 Core compatibility with DDI Codebook_2.docx - This document describes topics addressed at the meeting with Larry regarding Codebook compatibility for the DDI4 Core. It also gives an overview of classes and packages moved to the Legacy folder of the latest version of the model in EA.

To follow up the work she will look into possible to use Gantt diagrams tools to track dependencies.

Follow-up tasks

Working documents for internal review by the Sprint team developed at the Dagstuhl Sprint (week 1) are available from the Dagstuhl Sprint Task pages. Arofan is currently preparing the deliverables from the Sprint for presentation for the EB.

The following agreements regarding  follow-up tasks (1 - 11 below) were made at the meeting:

1. Overview and Purpose (document)

2. The Upper Model for DDI 4 Core (diagram and document)                                                                                             

           Arofan’s draft document is published for review by the group. Larry has already commented. Others to comment.

       3. Foundational Classes - concepts, variables, categories, classifications, etc. as presented in the Prototype, but updated to agree with the current modeling guidelines (diagrams and document)

           Dan G. has drafted a detailed document. Arofan will use this as a basis to prepare the Dagstuhl deliverable. Group to comment. Dan G. will provide a high level document with examples will be available for the January release.

       4. Design Patterns in DDI 4 Core – an introduction to the design patterns used (diagrams and document)

           This task presupposes integration of the new/revised patterns in the model. The document should be based on the final version of Patterns for the model. Some post-Sprint discussions may be required for the finalization. See for example DMT-294.

        5. The DDI 4 Core Process Model (diagrams and document)

             Jay is currently working on two different documents. One of the documents represents a description. The other document is about how to use the Process model (Example View with basis in the work of the Alpha network).

             He will liaise with Flavio while the week and get back to the group.

         6. The DDI 4 Core Data Description Model (diagrams and document)

a. Core classes

b. Unit-record data

c. Tall/skinny data

d. Cube structures for dimensional data

e. Key-value/No SQL data

               Documents for all a. – e. has been provided by Larry. Arofan has reviewed the documents with approval. Other group members are asked to review as well.

          7. DDI 4 Core and External Standards (GSIM, GSBPM, DCAT, DDI-C, DDI-L, etc.) [NICE TO HAVE]

                a. Use of external standards for DDI users

                b. Use of DDI 4 Core for external users

                c. Mapping to DDI Codebook

                Arofan has produced a document that will be published on the Wiki when refined. Arofan will liaise with Jay regarding input to this from the second Dagstuhl week. A goal is to publish this document before the TC meeting in Minnesota next week.

          8. Functional Views in DDI 4 Core (exemplary views) (diagrams and document) [NICE TO HAVE]

                a. Data Management View, encompassing process/provenance and data description.

                b. Data Description View, covering the straight data descriptions case(s)

                c. External Standards View, showing how DDI can be used in conjunction with an external domain standard [specific example TBD]

                 Views are likely to be a little different than described in the agenda. Views are currently viewed as diagrams and related examples.

                 Proposal for Views:

-Data Management View (including basics of DataDescription) – An idea is to base this on the work of the ALPHA network. Include a collection of diagrams rather than a single diagram.

-Process/Provenance

-External Standards View – this is now likely to be more about mapping.

-Time series - Would it be better to include this as a View rather than as a separate data structure?

           9. Canonical XMI for DDI 4 Core (valid XML XMI describing the DDI 4 Core Model with documentation).

               A draft is available. Needs update along with forthcoming model updates and some clarifications regarding associations, names and roles. Achim is in dialogue with developers of the XMI specification and will feed outcomes of the discussions back to the group.

          10. XML Schema Definition (XSD) for the DDI 4 Core Model (XSD schemas with documentation)

               Achim will work with Oliver  on the generated documentation and the XML schema. Will feed back in a two weeks time.

           11. OWL/RDF Syntax Binding for the DDI 4 Core Model  (RDF vocabulary definition and documentation) [NICE TO HAVE]

                 RDF will be drafted for the January deliverable and the public review. Choices regarding RDF will not impact on the PIM but rather on the RDF PSM. Achim reminds that the usages of ‘trace’, ‘refine’ and possible ‘derive’ in the model should be expressed.

                 This is useful for the transformation of PIM to RDF PSM as well as for for external standards mapping.

Model freeze mid November

It was agreed by the group to freeze the structure of the model by mid November, as much of the work including the Canonical XMI, the syntax representations, the EA diagrams and examples as well as the documentation and reporting work is depending on a stable model. All classes, names, associations, properties and cardinalities should then  be in place.

Core Scope

While Dagstuhl a set of classes were moved from the Class library to a Legacy folder in EA (Table 2 in the ‘Dagstuhl discussion regarding DDI4 Core compatibility with DDI Codebook_2.docx’ document). The trimmed down version of the DDI4 Core will form basis for the January deliverable. A complete view of what will be in the model  should be available by mid November.

Virtual MRT meeting 2019-09-18

Agenda as of email to srg list of 2019-09-18

1. Admin
   
   1. Dagstuhl stuff?
   2. New teleconference platform - options?
2. Production framework update/status
3. Process model update
4. Data model update
5. Other modelling topics

Minutes from MRT meting 2019-09-18

Attendees: Achim, Arofan, Dan G., Flavio, Hilde, Jay, Larry, Oliver, Wendy

Apologies: Jon

1. Admin

a. Dagstuhl stuff

Achim, Arofan and Hilde will have an organising call on Monday 23rd.

b. New teleconference platform - options?

Wendy explains that alternatives like Local zoom, Blue Jeans etc. would not fit the teleconference needs of the MRT group. The MRT group needs access to use the Zoom platform of the DDI Alliance. Arofan will ask Jared to provide the necessary details.  

2. Production framework update/status

Achim is working on the Canonical XMI from Enterprise Architect (EA). The goal is to have this ready by Dagstuhl. Achim will send the Canonical XMI out to the group when ready. People with EA can then try it. The Canonical XMI dialect used is the same as in the Canonical XMI from Lion.  It needs to be run through representations script, and be run on real examples.

The Canonical XMI work will serve as input to the forthcoming TC meeting as well, where input and output from COGS will be addressed.

3. Process model update

Arofan and Jay have started to write up the introduction to the Process part of the model. Flavio will review this. A goal is to have the first draft of the introduction document ready for Dagstuhl.

Jay has additionally been working on the draft of the upper model. He and Arofan will draft an introduction to the upper model too, before Dagstuhl.

4. Data model update

Flavio is working on the model integration. Key/Value has been added. The Datum part was not finalized before the meeting. A goal is to have this ready and integrated in the unified model before Dagstuhl. The integrated model then needs to be reviewed by the group.  

5. Other modelling topics

How to describe the model

Arofan explained that when writing up a specification the perspective needs to be top-down, starting with the upper model and then drilling down to the details. Larry and Dan G. expressed the importance of examples and use-cases when describing the model to the audience. Both perspectives seem to be important.

Standards alignment

In his diagrams Jay ‘traces’ classes from other standards. This makes it pretty clear how DDI4 relates to those standards, like for example PROV-O and GSIM. A standards alignment document would be needed in addition.

A version alignment document (DDI4 Core to other versions of DDI) is also needed. This will also be addressed at the TC meeting. Any output from the MRT work could feed into that.

Updating of foundational stuff

Concepts, Classifications, Variables etc.  needs to be reviewed and updated at Dagstuhl.

Version of EA to use at Dagstuhl

It was agreed to use EA v14 at Dagstuhl. Five of the participants will bring laptops with this to Dagstuhl.

Version control system

A version control system of the EA files should be available for Dagstuhl. Achim will look into this and liaise with Flavio about a good solution for managing and integrating files worked on by different people.

Forthcoming work before Dagstuhl

• Organising call (Achim, Arofan, Hilde)
• Canonical XMI (Achim, tests by the group)
• Modeling draft of Datum (Dan G., Flavio, Larry, Jay)
• Upper model draft; introduction document (Arofan, Jay, Flavio)
• Process model introduction document (Arofan, Jay)

Virtual MRT meeting 2019.09.11

Agenda (as of email to srg list of 2019.09.11):

1. Admin
2. Discussion of agenda/publication package and prioritization for the Sprint
3. Prouction framework clarification and status/RDF binding
4. Modeling update/status
   1. Data model
   2. Process model
   3. Upper model
   4. Foundational classes from prototype

Meeting minutes virtual MRT meeting 2019.09.11

Attendees: Achim, Arofan, Dan G., Jay, Hilde, Larry, Oliver, Wendy

Apologies: Flavio, Jon

1. Admin

Arofan sent the draft of the agenda for the Dagstuhl Sprint to the AG for their comments (post meeting 2019.09.11). The AG /Jared forwards the agenda to the EB.

2. Discussion of agenda/publication package and prioritization for the Sprint

Arofan will prepare a rough draft of the introduction document to the DDI4 Core specification and a description of the upper model for the Dagstuhl workshop.

Flavio war not able to attend the meeting, but continues the model integration work.  The Data Description parts have priority, thereafter the Process part that Jay is working on. 

The integrated model from Flavio will serve as basis for the detailing of the Dagstuhl work tasks and the way forward with the specification. It is important to end up with a product that has real functionality and can be extended later on.

Jay is working on a mash up bringing in different parts of the model, combining classes and looking into how other standards could be used. It would be good to have this available for Dagstuhl if possible.

3. Production framework clarification and status/RDF binding

The Canonical XMI is agreed by the group as the single source of truth. Currently modeling is done in EA that outputs a variant of XMI that need adaptation to comply with Canonical XMI. Achim is working on a crosswalk from EA XMI to Canonical XMI.  The goal is to have this before Dagstuhl, if possible by the end of the week. When ready Oliver can use the resulting Canonical XMI  to produce the representations.

It was discussed whether COGS can be used for the documentation (alternatively Swinx). Oliver explained that COGS would need some adoption to be able to input and output Canonical XMI (TC work). Possible XMI work related to intermediate testing of import of Canonical XMI to/export from COGS would need to come later on.

A discussion about an RDF representation as a DDI4 Core deliverable took place at the meeting. Achim expressed the need for the RDF representation to make use of other vocabularies in order to be used. He also pointed out that a review of the RDF, as well as a mapping of the DDI4 RDF representation to other RDF styles or schools is needed. Expert input will be needed to produce a final version of the RDF vocabulary for the DDI4 Core.

It was agreed that a preliminary RDF for discussion and review will be made available for the DDI4 Core deliverable for the end of the year.

4. Modeling update/status

It was not possible for Flavio to attend the current meeting. An update of the model was posted in Jira by Flavio post meeting (2019-09-12).

Weekly modelling meetings continue. Arofan asked the breakout group (Dan G., Flavio, Jay and Larry) to get back to him with feedback from the meeting on model integration of 2019-09-12, to be presented for the larger group.  

Arofan reminded that foundational classes and updates made to the Collection Pattern also need upgrading to fit the current modelling style.

It would be good if as much as possible of the modeling is done before Dagstuhl. At Dagstuhl EA will be used for modelling using the same version of the tool. A storage and version control system for EA files (eapx) needs to be in place before Dagstuhl (open source subversion repository/minimum alternative would be a Bitbucket repository). One gatekeeper will take care of the integration while other people can make edits and tests on working files.

Forthcoming work

Modeling/integration work continues.

Achim continues to work on the crosswalk between EA XMI and Canonical XMI. He will meet with Flavio to discuss questions and options regarding the modeling choices and solutions for the Canonical XMI. Oliver will also take part in this meeting.

Virtual MRT meeting 2019-09-04

Agenda (as of email of to srg list 2019-09-03)

1. Admin
2. Dagstuhl Work Agenda/Deliverables - agree final draft
3. Prouction framework status
4. Modeling update (Flavio)
5. Aligning fundamental structures
6. Cube data features to include - general data model integration
7. High-Level Model Update (Jay)
8. Status of process model

Minutes DDI4 MRT Virtual meeting 2019-09-04

Attendees

Arofan, Dan G., Flavio, Jay, Hilde, Oliver (part of the meeting), Wendy

Apologies

Achim, Jon, Larry

2. Dagstuhl Work Agenda/Deliverables - agree final draft

Arofan presented an updated version of the agenda for the DDI4 Core workshop in Dagstuhl. Comments from group members at the meeting and after will be addressed for the version to be sent to the AG and further to the EB.

4. Modeling update (Flavio)

Flavio has been working to compile the Cube/multidimensional part of the model. A draft is available post-meeting (2019.09.05). The next step is to integrate Long (Tall/Skinny) and thereafter KeyValue. The integration of Key Value is scheduled to before Dagstuhl (one to two weeks of work). The integration of the Process model will take place after KeyValue.

5. High-Level Model Update (Jay)

A new version of the high-level is available. The high-model will need updates along with changes to the integrated model. Arofan points out that it is important to express clearly what the entry points are to the model. When people come from the SDMX or from RAIRD for example, it is important that people are able to recognize themselves in the landscape of the high-level model. Dan G. points out the need of mapping between vocabularies. Naming rules are also important (how to deal with classes with identical semantics but different names in DDI4 and other standards, or with identical names but deviating semantics).

Jay proposed to provide a ‘source’ class for each class for each model diagram. The source could class be a DDI class from earlier versions, a GSIM class etc.

6. Status of process model

Jay has updated the process model. His proposal is less deterministic that that of the prototype, and unnecessary inheritance has been removed. Arofan and Jay will have a take on the process model before handing over to Flavio for integration.

Forthcoming tasks

Arofan follows up with the group regarding the agenda for the Dagstuhl Sprint.

He will also contact Achim and Oliver regarding the production framework and the Canonical XMI.

The modeling and integration work continues with the aim to have as much as possible ready before the Dagstuhl Sprint.

Virtual MRT meeting 2019-08-14

Agenda (as of email of to srg list 2019-08-13)

1. Admin: (if any)
2. Marketing discussion (branding) - report (Arofan)
3. Overview of proposed high-level model (Jay)
4. Status update on modelling (Flavio)
5. Status update on data doc (Larry)

Minutes DDI4 MRT Virtual meeting 2019-08-14

Attendees

Arofan, Dan G., Flavio, Hilde, Jay, Larry, Wendy

Apologies

Achim, Jon, Oliver

2. Marketing discussion (branding) - report (Arofan)

Arofan and Wendy took part in the meeting of the Marketing and Partnerships group. DDI 4 Core branding was discussed. Arofan will set up a list of proposals for a new name based on suggestions from the MRT group. He will send around the list to the MRT group for agreement before sending it to the marketing group. The name needs to be decided before the DDI4 Core is published for review. This is planned as the first release of the DDI4 prototype, not of the whole of it. The name should preferably reflect this.

3. Overview of proposed high-level model (Jay)

Jay presented his ideas of a high level model of the DDI4 (developed in EA v14, see post-meeting email from Jay to the srg list of 2019-08-14). This includes process and data documentation that is currently within the scope and timeframe of the Core, as well as parts of the prototype to be addressed by the MRT post-core.

The background for Jay’s ideas for a high level model for DDI4 is based on follow-up work from last year’s interoperability workshop in Dagstuhl, with aims to create data in the context of the Sendai Accord. This led the team to take an inventory of data pipelines regionally, nationally and internationally on the one hand, and a description of the data products associated with those data pipelines on the other. The team was also asked to think in terms of data science and algorithms to be used in order to get the most out of the resources and the data that was available.

The result represents a way to put together constructs that people in the ALPHA network use in a high level model that provides flexibility using non-deterministic control strategies. The prototype model is also used as library to produce mash up diagrams for different purposes and audiences.

Flavio and Hilde commented for example that ControlConstruct elements like Sequences IfThenElse are now enumerations rather than classes and thus platform specific. Hilde asks what this would mean in terms of inter-operability and reuse of construct related components – for example Sequences.

Arofan proposed that the high level model in some form and level of detail could be of help in order to describe the scope of the technical specification of the DDI4 Core in relation to the full scope of the DDI4, and to prepare a roadmap for forthcoming MRT work. What to include and what is for the future in relation to the Core? What to do with what is in the prototype etc?

Arofan also stressed the importance to scope the DDI4 Core specification in relation to the other DDI products like Codebook, Lifecycle and the DDI4 prototype.

Larry asks how much of Jay’s model that will be covered by the Core deliverable. A question is how to deal in the short and long term with content that is currently not scoped for the Core. In this context Larry referred to two Jira issues he recently posted that regard correspondence with DDI4 and Codebook: DMT-283 and DMT-284.

Flavio argued that it is important not to broaden the scope of the Core much due to the anticipated workload within the scheduled timeframe.

Status: Jay sendt around related EA 14 files (can be viewed in free EA Viewer) plus a zip archive with related pictures of the diagrams for the MRT to look at and evaluate.

Ongoing work/next steps

Work on modeling and data documentation continues and regular meetings are held

 A high level draft of how the Core spec is organised is to be outlined

Virtual MRT meeting 2019-08-07

Agenda (as of email of to srg list 2019-08-06)

1. Admin: work agenda plans
2. Marketing update (Arofan)
3. Model integration (Jay/Flavio)
4. Data document update (Larry)

Attendees

Arofan, Dan G., Flavio, Hilde, Jay, Larry, Wendy

Apologies

Achim, Jon, Oliver

Minutes DDI4 MRT Virtual meeting 2019-08-07

1. Admin: work agenda plans

Arofan will draft an agenda for Dagstuhl while the week and send to the MRT group for comments and agreement. The proposed agenda should then be sent to the Advisory Project Management Group (AG) for comments before forwarding to the EB.

2. Marketing update (Arofan)

Arofan has received an invitation from Barry in the DDI Marketing and Partnership Group to take part in a meeting about branding of the DDI4 Core on Monday August 12^th.

He pointed out that a possible new name for the DDI4 should not be too ambitious and not exclude extensions to the DDI4 (for example of Data description and Qualitative) after the Core has been finalized.

Alternatives for new names for the DDI4 Core were discussed. Some of the alternatives mentioned were ‘DDI Interoperability’ to indicate the connection with the FAIR principles, ‘DDI Cross-Domain’ to indicate that the DDI4 Core is not just for social science, or ‘DDI Data Hub’ (see https://en.wikipedia.org/wiki/Data_hub)  focusing on the power of DDI4 Core to handle various types of data from multiple sources. Interoperability was turned down by the group as the earlier versions of DDI facilitate interoperability as well. It was also discussed if a possible new name should be followed by a Core (for example ‘DDI Cross-Domain Core’) to indicate that the Core represents the first step in finalizing DDI4.

Status: Arofan and Wendy will discuss DDI4 Core branding further with the Marketing and Partnership Group at their meeting on Monday 12^th.

3. Model integration (Jay/Flavio)

Jay has been working on Process modeling and a paper for Modern Stats. The ‘process model’ is split in two. Data Management is intended to support data flow descriptions. Instrument Management supports the description of questionnaire flows. So far only the Data Management model has been developed, relying on a series of design patterns. Datum is turned into a pattern in this new proposal. The Signification pattern is left out and a new way to model transformations between tall/skinny has been proposed.

A side comment from Hilde was, as Instrument Management process modeling was mentioned, that the Data Capture package of the prototype is not yet final and would need a take - a suggested post-Core activity.

Status: Jay will prepare his work on Process for review by the MRT group as a whole. The process model(s) will subsequently be tied to Flavio’s work on the Core.

4. Data document update (Larry)

Larry has received content from Dan G. regarding multidimensional/cube data for inclusion in the document. This task has a dependency challenge as the EA diagrams for inclusion in the document are time consuming to make and work of importance to the integrated model is still ongoing.

Flavio pointed out the importance to focus on the Datum in the data description of the DDI4 Core.

Other

A modeling meeting will be held on Thursday August 8^th, directly after the TC meeting. Arofan Dan G., Flavio, Jay, Larry and possibly others will take part in the meeting. Flavio will put up an agenda in advance of the meeting (included in email to srg list of 2019-08-08).

Some possible topics for the meeting were mentioned:

• Ongoing modeling tasks/integration of work.
• Maintenance of EA files, how to keep track of different variants and versions, model integration (Where, how, who).
• Model tidy up. Make classes and relationships distinct and avoid duplication of properties.

Other proposal for upcoming tasks:

• Documentation tidy up. Make sure that terms and definitions are clear and understandable, and consistently used (How, who, when).
• A glossary of terms ((How, who, when).

Virtual MRT meeting 2019-07-31

Agenda (as of email of to srg list 2019-07-30)

1. Admin: Work agenda for Dagstuhl (for EB)
2. Production framework status update, new requirements, issues (Oliver)
3. Modelling update: Proposal for Patterns and Diagrams (Arofan/Jay/Flavio)
4. Process model update (Jay)
5. Data model document update (Larry)
6. Standards alignment update (Jay)

Minutes DDI4 MRT Virtual meeting 2019-07-31

Attendees

Achim, Arofan, Dan G., Flavio, Hilde, Jay, Larry, Oliver, Wendy

Apologies

Jon

1. Admin: Work agenda for Dagstuhl (for EB)

Work agenda for Dagstuhl:  Work on the agenda will start and need to be completed in good time before the workshop.

Vacations: Achim and Oliver will be on vacation from August till the beginning of September. Hilde will be off last half of August.

Video conferencing tool: The TC will test out Zoom as a replacement for GoToMeeting as video conference tool. This will not affect the meetings of the MRT who continue with GoToMeeting for now, till the TC has made a final decision on which tool to use in future.

2. Production framework status update, new requirements, issues (Oliver)

Background info: Email to the srg list of Oliver via Wendy, of 2019-07-31 (included in appendix point 1).

Oliver has transformed the latest canonical XMI from the Core selection from the last Sprint meeting into XSD. After correcting some uniqueness violations he had a validating schema file.

One example is ‘takesPlatformSpecificSentinelValues’ object relationship of InstanceVariable that additionally is inherited from RepresentedVariable. The object relationship is duplicated, and the information from the classes inherited from is overridden.

The violations detected by Oliver should be solved on the level of the model, rather than on the level of the level of the schema. The issues need to be looked further into.

Flavio will look into this and identify situations where duplication exists in connection with inheritance vs. situations where duplication not related to inheritance exists.

Larry suggests to have a separate modelling meeting to solve this and other outstanding modelling issues (see below).

Status: Agreed

3. Modelling update: Proposal for Patterns and Diagrams (Arofan/Jay/Flavio)

Arofan pointed out that progress on the modeling is dependent on how to deal with patterns.

The proposal includes the following:

A simple heart of the Core deliverable should include the model in Canonical XMI and documentation with diagrams:

• One main variant of the model in Canonical XMI without the patterns, accompanied by descriptive diagrams where the patterns are not shown
• Another variant of the Canonical XMI for developers and advanced users where the patterns are kept in, accompanied by descriptive diagrams where patterns are kept in.

The background for this proposal is that some user groups find the patterns confusing.

Views should be documentary, each sub-set (View) relating to a particular use-case.

A discussion followed regarding the role of the diagrams, notes in diagrams, packaging and how to deal with the ‘realize’ association in the prototype on the short and the long term.

• Code reuse: Dan G. pointed out that design patterns supports code reuse and that it should be conveyed that code reuse is part of the design.
• Diagrams/crosswalk between variants: Flavio proposed that diagrams documenting the main variant of the model could be produced by hiding pattern related classes and associations in EA. The crosswalk between the two variants of the model could be done by removing the patterns in the ‘main’ model XMI variant.
• Packaging: Achim pointed out that it can be risky to do transformations between the two variants with Enterprise Architect, and will look into if a way of packaging to avoid this would be possible. (See explanation and results of first tests at point 2 and 3 in the Appendix).
• Notes in diagrams and Canonical XMI: Larry asked if notes in diagrams could be represented in Canonical XMI. There was a discussion and Achim agreed to look into this. See explanation and the results at point 2 of the Appendix.
• The ‘realize’ association: The ‘realize’ association for incorporating patterns from the prototype needs to be dealt with. Arofan proposes to keep the ‘realize’ association in for the first release of the Core due to time constraints. Achim points out the need to take on this, preferably on the model level, or at a minimum to post-process the XMI (see agreement below and the Appendix).

Agreement regarding release products: The group agreed that the products to be released would be:

(1)  A "user" model expressed in Canonical XMI, accompanied by diagrams and explanatory text, which would not include the design patterns

(2)  A "developer" model which would be the model expressed in Canonical XMI, accompanied by diagrams and explanatory text with the patterns in place

(3) Functional Views expressed as diagrams and explanatory text for a specific functional purpose, not accompanied by Canonical XMI or other machine-actionable representation at this time

Further, the current style of incorporating patterns - the "realizes" associations need to be removed in the XMI.  Either by another approach in the model (preferred) or by post-processing of the XMI.

Please note that questions regarding how packaging and transformation to generate these products remain open, but that some practical ways of doing this from the EA working file have been identified. More exploration here is being undertaken. The idea here is to not restrict additional functionality/products in the future, but to come up with a useful set of outputs for the 2019 release. Additional machine-actionable products may be identified and produced going forward, etc.. 

Follow-up work

Flavio, Jay and Larry will organize modelling meetings in the coming weeks. Dan G. will take part in some of the meetings.

Appendix

1) Email to the srg list of Oliver via Wendy, of 2019-07-31

Hi,

 I transformed the latest canonical XMI from the core selection into XSD. After correcting some uniqueness violations, I had a validating schema file.

 The violations are:

CategorySet -> Contains, IsStructuredBy

InstanceVariable -> BasedOnConceptualVariable, TakesPlatformSpecificSentinelValues

ConceptualInstrument -> Name, Usage

InstanceQuestion -> Name

GeographicUnitClassification -> References

GeographicUnitTypeClassification -> References

ClassificationItem -> Denotes

Code -> Denotes

SamplingDesign -> ExpressesAlgorithm, SpecifiesGoal, ImplementedBy

SamplingProcedure -> HasDesign, IsExpressedBy, ComponentMethodology

ConditionalControlStep -> Executes

WorkflowStepSequence -> Name, Purpose

 Best,

Oliver

2) Email from Achim to srg list 2019-07-31 (post-meeting) :

I made a couple of tests. The results support the ideas we were discussing in today’s meeting.

I use the latest version of Enterprise Architect, 15.

Notes in diagrams:

Notes can be independent note boxes in diagrams. They can have a “usage” relationship to a class or an association (it is not possible to connect the relationship to a property). A note can have “usage” relationships to multiple items.

The notes appear in the regular section (not EA extensions) of XMI as owned comments which have one or more annotated elements. If the note has not relationship in the diagram, the comment in the XMI would have the package as annotated element where the diagram lives in.

This means the notes in the diagrams would be maintained in the XMI and the related Canonical XMI. This is independent of the diagrams which are expressed in the EA extensions of XMI.

Design patterns

The design patterns are organized in packages (like in the prototype). These packages could be all in one master package “DesignPatterns”. This package can be exported in EA as separate XMI file. EA will then put then the package in a model “EA_Model”.

All other packages could be all in a master package like “PIMforUsers”. Again, this package can be exported in EA as XMI to a separate file. This file can be transformed into Canonical XMI.

This means that the idea of separation of concerns would be supported. The design patterns for the model developers and the advanced users on one side and on the other side the model.

The EA file (EA model) would comprehend everything. The two master packages make the separation. The two exported XMI files would be separate models.

The open question is still what could be done with the “realize” association.

Achim

3) Email from Flavio to srg list 2019-07-31 (post-meeting), response to Achim’s email:

Achim,

I've just tested the XMI by package you suggested and it seems to take care of the "realizes association" removal. I will confirm when we have the full package structure in place, but it looks promising.

Flavio

Virtual MRT meeting 2019-07-24

Agenda:

1. Alliance admin update: Dagstuhl attendance discussion
2. Production framework status update (Oliver)
3. Modelling update (Flavio)
4. Data model document (Larry)
5. Process model update (Jay)
6. Standards alignment update (Jay)

MRT meeting minutes 2019-07-24

Attendees

Achim, Arofan, Dan G., Flavio, Hilde, Jay, Larry, Oliver, Wendy

Apologies

Jon

1. Alliance admin update: Dagstuhl attendance discussion

The group that will attend the Dagstuhl workshop in person is Achim, Arofan, Dan G., Flavio, Hilde, Jay and Larry. Oliver and Wendy will attend virtually. Oliver will be available to meet virtually and take on MRT tasks on most days, possibly except Thursday. Wendy would be available on afternoons Dagstuhl time on Monday through Wednesday.

This is a small group of people of similar size as when DDI3 was finalized in 2007.

The work plan for the meeting needs to be ready good time in advance of the meeting, like in the first or second week of September. Forthcoming documents on DataDescription (Larry), Process (Jay) and Foundational will help in the understanding of what remains to do. Arofan also pointed at the need to review the field level documentation.

Delivery of a DDI4 Core specification will happen immediately after the Sprint. The model in Canonical XMI needs to be final by the Thursday of the Dagstuhl week. Arofan raised the question if the finalization of actual bindings/representations would need to be ready by Dagstuhl as well, or if this could wait till the week after if fixes on the model need to be made at the Sprint. Wendy expressed that it should not be an issue to delay the finalization of the representations/bindings for a week as Oliver who will do the push will not be present at the Sprint.

Wendy asked if the group wanted to have (read only) access to Lion for the Sprint, including listings of what is in the views (read only). The group wanted to have this, and Wendy will make sure that Lion will be available for the Sprint. Wendy also has copies of DDI4 Drupal content of the Prototype, the Core components, active development work and the dump from DDI 3.2.

EA v 14 will be used for modelling work at the Sprint. Most participants have this or will upgrade their EA versions before the meeting. The Canonical XMI will be based on this.

Status: All agreed

2. Production framework status update (Oliver)

The implementation of agreements made at the last MRT meeting has been made. This means that no flattening is done and inheritance by extension to abstract classes is used.

Oliver experienced some smaller issues with ‘realize association’ from the prototype resulting in duplicates. This is different from UML realization.

Further validation will take place when the model or parts of the model is ready to test.

3. Modelling update (Flavio)

Flavio will start to prepare the patterns first and aims to have something ready for next week. Jon will set up an environment in Bickbucket where the files from the Core can be maintained.

Patterns are useful to keep the model consistent. Arofan asked Flavio and Achim to explore and discuss different options regarding how to deal with patterns and get back to the group with a recommendation. Jay will input to this work.

Achim explains that in the prototype copies of properties and associations are made by hand and cannot be understood by UML tools. This results in multiple inheritance (mostly from identifiable).

Alternatively UML inheritance could be used, but there could be issues with that as well. Properties may for example be renamed in an applied sense and it is not clear how tools deal with this.

UML realization (see https://www.uml-diagrams.org/realization.html) might be a third alternative.

Flavio and Achim will liaise regarding how to deal with the patterns and will get back to the group with a proposal.

4. Data model document (Larry)

Larry has made some updates to the Data model document, mainly by breaking up the larger diagram.

5. Process model update (Jay)

Jay is working on the process model. He will liaise with Larry and send something around.

6. Standards alignment update (Jay)

Arofan asked Jay to put standards alignment on wait for the moment.

Virtual MRT meeting 2019.07.17

Agenda (as in email to the srg list of 2019.07.16)

1. Alliance admin news update, if any
2. Production framework status update (Oliver)
3. XML Binding Issues update (Achim)
4. Modelling Issues (Flavio):
   • Update on data model integratio
   • Patterns issue
   • UML data types issue
5. Standards alignment update (Jay)

Minutes DDI4 MRT Virtual meeting 2019-07-17

Attendees

Achim, Arofan, Dan G., Flavio, Hilde, Jay, Larry, Oliver, Wendy

Apologies

Jon

1. Alliance admin news update

The invitations for the Dagstuhl workshop have been sent out. The deadline to respond to the invitations is July 23^rd. Work to prepare the meeting agenda will start soon.

2. Production framework status update (Oliver)

Oliver will add the abstract classes in the PSM and extend from that pairing. Abstract types can be used in the related schema (see also minutes from MRT meeting 2019-07-10).

It was discussed whether the PSM is needed as the flattening of the model (which will no longer happen)  according to Oliver was the only thing that happened at that step. Arofan comments that the distinction between PIM and PSM might be useful in the future and suggests to keep both for this reason. Achim points out that the PSM conceptually and structurally should reflect both the syntax of the representation, so that any changes to the syntax should be reflected on the level of the PSM.

Oliver will make the needed changes to trivialize the PIM into PSM that will form basis for schema with abstract classes in. This work will be done in the next week or two.

Possible overriding (for example when it comes to cardinalties) in the model and the corresponding representations  should be looked into and removed. Inheritance should occur by extension.

               Status: Agreed

Jay asked what the rationale for the proposal to flatten the model on the level of the PSM. Arofan replied then that it was because of the long inheritance chains, but that this has been improved so that flattening would not be needed any more. Some patterns implementations may potentially complicate this and would need to be looked into.

3.  XML Binding Issues update (Achim)

Achim has started to produce a generic mapping table. This is complicated but useful work. He will have something to show to the group possibly next week.

4. Walk-through of revised Data Description document

Larry is currently revising the Data Description document from the NADDI sprint by adding content and including snippets from the model related to the different data types described. At the meeting he walked the group through changes as of the meeting day. Work on the document will continue and will incorporate changes in the model. The multidimensional/cube proposals from Dan G. has not yet been included.

It was discussed whether the description of the Classification piece, process and Patterns and Provenance should be included in the same or in a different document for the DDI4 Core deliverable. A goal could be to have a draft of all available by the end of the Dagstuhl week.

4. Modelling Issues (Flavio):

Flavio pointed out that the following needs to be addressed. Some of those are:

• Finalising the modeling guidelines
• Clean up of the documentation
• Revise the definition of Datum
• Decide if the Signification pattern should be kept or not
• Update class documentation

Achim and Wendy proposed to use an editable markdown template to prepare the documentation of classes, properties and associations. In the end the markdown can be copied into the relevant comments fields of the EA. Flavio will coordinate this work.

A sandbox is needed to store different work products. Flavio will ask Jon about which platform to use for this.

Virtual MRT meeting 2019-07-10

Agenda

1. Alliance admin news update
2. Production framework status update (Oliver)
3. XML Binding Issues (Achim)
4. Modelling update (Flavio)
5. Standards alignment update (Jay)

Minutes DDI4 MRT Virtual meeting 2019-07-10

Attendees

Achim, Arofan, Dan G., Hilde, Jay, Wendy

Apologies

Flavio, Jon, Larry, Oliver

1. Alliance admin news update

Achim informs that invitations to the Dagstuhl workshop will be sent soon.

3.  XML Binding Issues (Achim)

Achim and Oliver have been meeting regarding how to approach XSD schema design.

In the follow-up Achim, Oliver and Arofan will discuss the lower level xml schema issues.

Higher level issues like use of abstract classes and inheritance in the PSM and the corresponding representations were discussed at the meeting, see below.

3.1) Can abstract classes be used in the PSM and the representations?

• Achim proposes inheritance by extension to abstract classes in the XSD. The model has chains of inheritance and this should be reflected in the XSD schema. An important step to shorten inheritance chains has been done on this when Identifiable has been moved from an abstract root class to a structured property. Further steps to shorten inheritance chains must also be solved on the level of the model.

 According to Wendy,  use of abstract classes in bindings have not created problems earlier.

• Multiple inheritance/Inheritance of the same classes from different sources: According to Jay there is a problem when the Process and Collection patterns are used in the same workflow. Achim explains that his needs to be solved on the level of the model. An idea can be to use single patterns.

Status: Achim’s proposal to use XSD abstract types for UML abstract classes was agreed. Oliver will be asked to implement the proposal for the XSD, keeping in mind that inheritance chains still can be long.

3.2) Cardinalities

Differences between cardinalities between UML and the XML schema have been detected. Wendy confirms this is an error.

Status: Achim will file an issue, see DMT-281.

3.3) Enumerations

This is used in connection with realization of concrete classes from abstract classes. This is often used, but some only allows a single value. Arofan and Wendy think this can be a workaround from abstract classes.

Status: Enumerations with a single value may be removed when XSD abstract types will be used.

3.4) PSM XMI

Canonical XMI can now be imported to Enterprise Architect but not yet in Magic Draw.

Status: Achim will follow up on this and make further analyses.

3.5) Model and representations mappings

A mapping table of model components to OWL and XSD for a restricted set of classes would be a good starting point for understanding the relationships.

The work can also inform the work on Standards alignment (see point 5. below).

Status: Achim will start the work on a mapping table for model and representation components.

Wendy points out that work is ongoing within the TC to look at XML bindings/representations for the DDI across versions. The current discussions in the MRT group could inform that work. The current discussion is going on on GitHub.

5. Standards alignment update (Jay)

On July 4^th Jay sent an update of the ‘Relationship to Other Standards/Interoperability’ document from the NADDI Sprint by email to the srg list. No comments were received by email before the meeting.

Jay’s approach regards to use the NADDI Sprint document on standards alignment as a basis and add content that regards DDI from a producer perspective. How does it look like when other standards uses parts of the DDI. DISCO would be one use-case.

At the meeting Hilde commented that a mapping similar to what Achim suggest for the representations would be a good first step to get an overview of possible relationships between the DDI and a selection of other standards. In her view focus should be put on the standards the MRT ‘committed to’ (DCAT, GSIM, GSBPM and PROV-O).

Wendy commented to this that other standards that deals with geography usefully couls be addressed as well.

Achim pointed out that a limited set of classes need to be explored for the different standards.

Virtual MRT meeting 2019-07-03

Agenda

1. Alliance admin news update
2. Confluence & Issues resolution update (Hilde)
3. Production framework status update (Oliver)
4. Modelling update (Collections, etc.) (Flavio)
5. Data description update 
6. Final package contents/milestones - brainstorm
7. Standards alignment - Jay to propose a paper

Minutes DDI4 MRT Virtual meeting 2019-07-03

Attendees

Achim, Arofan, Dan G., Flavio, Hilde, Jay, Larry, Oliver, Wendy

Apologies

Jon

1. Alliance admin news update

Achim reminded the group about the EB’s approval of the Dagstuhl workshop of 29^th September through 4^th October. He informed about the request of the EB that the group informs about expected deliverables before the Dagstuhl Sprint and report back directly after the Sprint about the outcomes and factual deliverables. Jared will send out invitations to the Dagstuhl Sprint.

2. Confluence & Issues resolution update (Hilde)

Hilde and Arofan will discuss the take on the Jira issues provided by Wendy. A meeting regarding this  needs to be scheduled.

3. Production framework status update (Oliver)

Achim and Oliver will meet in Cologne on Friday 5^th to discuss the production framework.

Oliver asks people to look at the schema from the last production run, sent by email to the srg list on June 5^th. Changes include things in attributes being put into child nodes.

4. Modelling update (Collections, etc.) (Flavio)

Flavio is waiting for an update to an EA programme version that can import Canonical XMI of the Core directly. Meanwhile he will work on EA file exported from Larry’s newer EA version (14.5).  

5. Data description update 

Arofan point out that the work on the Cube/Multidimensional and Key Value pairs should be put in as part of Larry’s work to consolidate things.

Dan G. will build the cube/multidimensional model in Visio and send to Larry. Dan G., Larry and Flavio will discuss and the model will be implemented in EA.

6. Final package contents/milestones – brainstorm

Arofan asks the group to think about what the DDI4 Core deliverable will look like, and asks if the DDI4 Prototype way of packaging a good way, or if things need to be changed for the deliverable of the Core?

-Achim pointed out that a good understanding of the deliverables is important both for the work of the MRT group itself, for communication with the EB before and after the Dagstuhl Sprint, as well as for users later on.

-Flavio expressed that the set of Views for the deliverable needs to be defined. Which Views should be included? Examples are description of concept systems, data management (as part of process) etc. This needs to be further explored.

-Arofan pointed at the importance of good documentation for users both high level and more detailed. The data description document is already in progress. Technical and field level documentation needs to be looked into.

-Larry mentioned that it would be useful to describe what the DDI4 Core will allow users to do, in comparison with earlier versions of the standard.

-Hilde focused of on the over-all understandability of the model. Understandable definitions and examples should be provided for components where this is currently. Naming rules used need to be made public etc.

The discussion regarding the DDI4 Core deliverables will continue in the forthcoming meetings.

7. Standards alignment - Jay to propose a paper

Jay will use Wendy’s paper on standards alignment from the Ottawa Sprint as a starting point for a proposed paper on standards alignment. According to Jay, what has been done till now is good but DDI centric.

It would also be possible to see things the other way around, like how to make DDI more accessible for other standards to use. Jay proposed that the DDI4 Core can be packaged in a way to make it accessible to other groups.

Arofan pointed out that the standards we ‘committed to’ are DCAT, GSIM, GSBPM and PROV-O. As Achim indicated in the last MRT meeting, this does not mean that everything from these standards needs to be brought into the Core.

Standards are expressed in many different ways. Larry expressed that alignment to other standards can be made without necessarily adopting the semantics of the relevant standards.

Jay will continue to work on the document produced at the Ottawa Sprint, make edits and add new content.

Standards alignment could, additionally, be proposed as one topic for the Dagstuhl interoperability workshop scheduled to October.

Virtual MRT meeting 2019-06-26

Agenda

1. Alliance admin news update
2. Confluence & Issues resolution update (Hilde)
3. Production framework status update (Oliver)
4. Modelling update (Collections, etc.) (Flavio)
5. Data description discussion: overall plan across all data formats - how do we move forward?
6. Standards alignment/other issues from last week

Minutes DDI4 MRT Virtual meeting 2019-06-26

Attendees: Achim, Arofan, Flavio, Hilde, Jay, Larry, Oliver, Wendy

Apologies: Dan G., Jon

1. Alliance admin news update

Wendy updated shortly about preservation of content from Drupal. A copy of the application is retained. This includes the following:

• Prototype - content sent out for Prototype review.
• Core - Selected core packages only.
• Active development work - core plus visible packages (those that were either being actively worked on or were included in future development plans at the time of the Prototype, for example Qualitative).
• Preservation set - all of Active development plus hidden packages of other development work (for example earlier versions of sampling and collections pattern) - does not include DDILifecyle 3.2 content or classes in general "DUMP" packages of individual classes that are not associated with any of the packages that were included in the Preservation set.

Funding: Achim reported from the latest meeting in the Excecutive Board that both Dagstuhl workshops for this year have received funding:

·      DDI 4 Core - Development of a Robust and Sustainable Model (September 29^th – October 4^th)

·      Interoperability of Metadata Standards in Cross-Domain Science, Health, and Social Science Applications II (October 6^th – October 11^th).

A DDI4 Core specification as well as delivery plans are required outcomes.

A proposed MRT workshop in connection with IASSIST has, on the contrary, not received funding. Nor has a request for RDF integration expertise.

2. Confluence & Issues resolution update (Hilde)

Hilde has started to link Jira issues from Wendy to MRT tasks (Prototype_Review, Post-Prototype and issues transferred from TC to MRT). She has provided Arofan with an overview of how she means they relate. She will liaise with Arofan regarding which issues are resolved or do not apply any more, and prepare them for discussion.

3. Production framework status update (Oliver)

Oliver has not had time to look into this since the last meeting. Achim and Oliver will liaise regarding the production framework.

4. Modelling update (Flavio)

Flavio is expecting a recent Enterprise Architect update that can handle the model in Canonical XMI. EA 13,5 and onward imports this correctly.

Achim informs the group post meeting that v. 15 of EA is now available.

5. Data description discussion: overall plan across all data formats - how do we move forward?

A team will follow up on this from the take at the NADDI Sprint and will feed back to the larger group with outcomes:

• Flavio - Integrate model pieces for the various data structures.
• Larry - Editorial.
• Hilde – Continuity from the NADDI Sprint; broader audience view.

6. Standards alignment

Arofan suggests that a limited set of standards for alignment for the MRT work is GSIM, GSBPM, DCAT and PROV-O (Possibly Dublin Core).

Achim suggests to relate to sub-set of the standards in terms of classes and ideas, and to focus on parts of the standards that are relatively simple and stable. Standards are different as some are expressed in models, some in UML, some as RDF and some as XML or RDF representations. Actual examples are needed to decide how alignments best could be done. Relationships to other standards will need to focus on specific version(s) of the standard.

Jay comments that perhaps some examples from Simon Cox about standards alignment to DDI4 could be used.

Flavio points out that if classes from specific standards are integrated in the Core it can be a challenge to follow up in updates in them.

Follow up work regarding standards alignment: The group to discuss this on the srg email list. See email from Arofan to the group in the appendix.

Appendix

Post-meeting email from Arofan to the srg list 2019-06-26

On today's call I heard several points about standards alignment which seemed to me to be potentially congruent:

(1) We should use a limited definition of "round-tripping" (this has not always been the case in past discussions)

(2) We should focus on specific parts of/classes from external standards, and not take on-board entire models

(3) Maintaining external models inside of DDI 4 comes with a potentially massive, non-scalable overhead which we probably cannot support

(4) A limited list of external standards for "alignment" (at least in documentary form) has been identified for delivery as part of MRT: GSIM, GSBPM, DCAT, PROV-O (and it sounded like Achim was thinking about Dublin Core, although we did not mention that in any documentation)

(5) Other people who are primarily using some different standards want to integrate with DDI, and we have some examples from Simon Cox (per Jay's comment). They want to use DDI as a vocabulary for describing data

(6) Different external standards may require different forms of alignment, depending on what they are being used for (the Sprint document shows what has been done in the past here)

This suggests that we might take a fairly minimalist view of our alignment with other standards, and avoid including in our model information which is not germane to describing data and related metadata/processing. What we would need to do is to indicate how we see our model being used vis-a-vis each external standard, and to possibly identify the technical mechanism from our bindings for references to things in other people's models (ie, include a place where URLS could be provided, with sufficient semantic so that processors are aware that a different standard is in operation at the far end. I think we already do this, but we could look at it to be sure).

Obviously some larger questions remain, Achim has suggested that we work to specific examples. Is it possible that we could take some of Simon's input (which Jay mentioned) and derive some examples from that? DCAT would be a good starting point as an example (I know Wendy was involved in the Dagstuhl group looking at this last year, although the work was only a rough draft.)

We may have at least two different standards-alignment audiences: DDI people who want to use other standards, and external people who want to use DDI. I suspect these two groups will have somewhat different requirements. It would make sense for us to think about how different these perspectives are, and whether we can satisfy both with a single approach.

Virtual MRT meeting 2019-06-05

Agenda

1. News from IASSIST members meeting/Alliance Admin (EB, SB, etc.) - Achim
2. Confluence & Issues resolution update (Hilde)
3. Production framework status update (Oliver)
4. Modelling update (Collections, etc.) (Flavio)
5. Data description discussion/update (Various)
6. Process model presentation (Jay/Flavio)

Minutes DDI4 MRT Virtual meeting 2019-06-05

Attendees

Achim, Arofan, Dan G., Flavio, Hilde, Jay, Larry, Oliver,

Apologies

Wendy, Jon

1. News from IASSIST members meeting/Alliance Admin (EB, SB, etc.) – Achim

Achim was not in Sydney and is awaiting feedback from Ingo and Jared. We hope next week some information will be available. An EB meeting is foreseen for mid-June. Achim points out that the agenda with linked meeting documents have been sent to different DDI email-lists. This way they are accessible for the MRT group members and others to read.

2. Confluence & Issues resolution update (Hilde)

Hilde asked Flavio to look into issues related to the Collection Pattern, as proposed by Arofan. While Flavio recommends to  close some of the related  issues, some issues in particular DMT-238, DMT-203 and DMT-189 will require further discussions.

Hilde is currently working to link Prototype_Review issues, Post-prototype issues and issues transferred from TC to DMT, to MRT tasks. She will liaise with Arofan next week regarding this work.

3. Production framework status update (Oliver)

Oliver sent a DDI4 build by email to the srg list on the meeting day. This includes a validation schema file produced on basis of the canonical XMI from the prototype release. A proposal for PSM was also included. Back-tripping is not yet possible.

Achim has looked into the former version of the build that Oliver provided and has some questions regarding the realization of the PSM. Achim and Oliver will liaise regarding this and will get back to the larger group with input from their discussions.

4. Modelling update (Collections, etc.) (Flavio)

In an email to the srg list of 06.01 Larry questions Flavio’s suggestion that aggregations would not need naming. Larry sees this as a problem, for example when describing XPaths where there are two aggregations with the same target type. Larry and Flavio will liaise to frame up the discussion and will get back to the bigger group with input.

5. Data description discussion/update (Various)

Multidimensional/Cube:

Dan G. and Flavio will get back with modeling input as quickly as they can.

Key/value pairs:

Flavio is preparing an example from streaming. He and Jay will get back with feedback regarding key/value pairs in one to two weeks.

Data Description with examples document:

Hilde will take on this again after the meeting with Arofan about the issues.

6. Process model presentation (Jay/Flavio)

Jay sent an updated version of his presentation of Process by email to the srg list right after the meeting. This proposal covers both white and black boxes.

The audience for the presentation is both the MRT group and ModernStats.

Jay asks people to look into this till next time and get back with comments.

He points out that there would be several ways to deal with this task, and that terms probably needs to be addressed.

Virtual MRT meeting 2019-05-29

Agenda

1. Any news regarding the EB or upcoming members/SB meeting at IASSIST
2. Issues resolution update (Hilde)
3. Production framework (including binding issue for XSD) (Oliver)
4. Applying the Collections model (Flavio)
5. Process model presentation (Jay)
6. Data description discussion/update

Minutes DDI4 MRT virtual meeting 2019-05-29

Attendees

Arofan, Dan G., Flavio, Hilde, Jay, Jon, Larry, Oliver

Apologies

Achim, Wendy

1. Any news regarding the EB or upcoming members/SB meeting at IASSIST

No news are available yet, but are expected after the Annual DDI meetings.

2. Issues resolution update (Hilde)

Hilde is looking into Prototype_Review and Post-Prototype DMT Jira issues, as well as issues transferred from TC to DMT, and assigning them to MRT tasks. Flavio will look into the issue related to Collections linked to the MRT task DMT-232 and will get back to Hilde regarding their status.

3. Production framework (including binding issue for XSD) (Oliver)

Oliver raises a question originally posed by Larry regarding some structured data types properties becoming attributes and some child nodes in the xml representations (bindings).

Larry points out that it is not a good idea to arbitrarily making things different when parsing into xml, and that child nodes in general are much easier to deal with.

Arofan expresses that this needs to be looked further into.

Oliver will, in the course of one to two weeks, try to draft a simplified set of rules for this, and at the same time produce a schema that implements the proposed for rules - all for review by the group.

               Status: Agreed

4. Applying the Collections model (Flavio)

Flavio sent an update of the collection pattern including an XMI and an EA file to the group on the meeting day (05.29). He expresses that the current proposal should not deviate too much from the original.

Some implementation examples are needed to check if things need to be fixed.

Arofan means that the current proposal could be applied as things are modeled. Revisions of the over-all the model can be done when the pattern is tested through application on different scenarios.

5. Process model presentation (Jay)

At the meeting Jay went through the slide deck on Process sent by email to the srg list on the meeting day (2019.05.29).

Jay’s ideas include a proposal for simplification of the Process model as in the prototype, focusing on pre- and post-conditions and supports data management.

Dan G. pointed out that with Jays proposal for Process, the Methodology pattern of the prototype might be redundant, to which Jay agrees.

Jon points out the importance of transparency of processes.

Arofan expresses that data lineage is the primary scope of Process for the DDI4 Core.

Jay will continue to work on the Process model. GSIM, GSBPM and PROV-O will be looked into.

Jay, Flavio and Jon will provide different pipeline examples to inform the further work.

Other

Dan and Flavio continue to work on multidimensional data.

DDI4 MRT Virtual meeting 2019-05-22

Agenda (as in invite of 2019-05-21)

(1) Update on production framework/tools (Oliver)

(2) Update on modelling guidelines (Flavio/Achim)

(3) Data Description

- Examples document

- Event data

- Cube data

- No-SQL data

Minutes DDI4 MRT Virtual meeting 2019-05-22

Attendees:  Achim, Arofan, Flavio, Hilde, Larry, Oliver, Wendy

Apologies:  Dan G., Jay, Jon

(0) Documents from the NADDI Sprint

Documents from the Sprint have been reviewed and approved by the group and will be published as deliverables. An overview and scope of DDI4 Core document has already been published.

(1) Update on production framework/tools (Oliver)

Oliver has sent the first PSM to Achim to verify that it is still Canonical XMI. Achim will try to validate the file. Oliver expects to finish this while this week.

(2) Update on modelling guidelines (Flavio/Achim); Collection Pattern

The modeling rules agreed at the NADDI Sprint will be turned into guidelines by Achim and Flavio.

Flavio has updated his version of the Collection Pattern according to the naming rules agreed at the NADDI Sprint (see email to srg list of 2019.05.22).

Hilde asked Flavio to provide a document that describes differences between his latest draft and the Collection Pattern model as sent by email by Larry to srg list 2019.04.10. She likes the understandability of the 2019.04.10 version.

A goal is to finalise Collection Pattern soon. Larry will provide some examples for testing.

It was agreed that Larry organizes a call with the Collection Pattern as topic, scheduled to Tuesday 05.28 at the usual meeting time. Achim, Arofan, Flavio, Hilde, Larry and possibly Jay will take part in that meeting.

(3) Data Description

Lively discussions regarding data description modeling issues have recently been going on by emails to the srg list.

A general point made by Arofan is that understandable entry points to Event and Cube etc. are critical, in order to make the model accessible for users without too much documentation. Larry points out that this is important but needs to be done in a way that the semantics of the model does not explode.

- Event data

Flavio and Larry have met, and a draft model of the Tall Skinny format (often used for events) was sent by Larry to the srg list 2019.05.17. Flavio sent a different proposal for event, based on key/values (including RAIRD) to the same list 2019.05.22. The two proposals don’t contain the same classes.

This needs follow-up discussions in the coming weeks. Arofan suggests that Flavio, Larry and Jay follow up on the proposals and try to bind them.

- Cube data

Flavio will follow up with Dan G. regarding Cube/multidimensional.

Flavio points out that the Dan G.’s BLS model makes it is hard to drill down from aggregate to microdata, and between different levels of aggregation.

Wendy points out that the model need to do as least as good as- or better than what is currently in Codebook and Lifecycle.

Arofan suggests that he, Dan G., Jay, Flavio and meet to follow up on this.

- No-SQL data

Flavio and Jay are working together on this.  

Way forward:

-Modeling guidelines document

Achim and Flavio will follow up on this

-Modeling/data description tasks

Flavio arranges the ongoing tasks when it comes to level of maturity as follows:

1) Collection Pattern

2) Tall Skinny

3) Key value

4) Cube/multidimensional

The Collection Pattern will be discussed next Tuesday as indicated above. The goal is to be able to agree a final version in the near future.

The next thing to be discussed will be cube/multidimensional. This can happen when Flavio has been in contact with Dan G. about the topic.

-Process

Jay has agreed to provide an explanation of the process piece.

-Alignment with other standards

Arofan points out that a general approach to alignment with other standards needs to be agreed.  

This is also important in connection with modeling of multidimensional data. SDMX and DataCube both have limitations when it comes to connecting to sources. MDX should be looked into and see what is there etc.

-Jira issues related to MRT tasks

Hilde will work on this in collaboration with Arofan.

-Other forthcoming tasks

Representations/representations, Citations, what do with Study etc. are tasks we will get back to.

DDI4 MRT Virtual meeting 2019-05-08

Agenda (as in invite of 2019-05-07)

(1) Report of the EB response to the Sprint outputs (if Achim is present)

(2) Approval/revision of Sprint deliverables and Overview document

(3) Status of Production framework

(4) Organizing post-Sprint work

A. Modelling issues: 

    (a) Finalizing the guidelines

    (b) Moving forward with Collection Patterns

    (c) Refining the production model from the Lion pull

B. Data description work: 

    (a) Completing the examples document; 

    (b) Multi-dimensional model; 

    (c) non-SQL (etc.) data; 

    (d) Event data model

Minutes DDI4 MRT Virtual meeting 2019-05-08

Attendees:  Arofan, Dan G., Flavio, Hilde, Jay, Oliver, Wendy

Apologies:  Achim, Jon

(1) Report of the EB response to the Sprint outputs (if Achim is present)

This will be detailed later on as Achim was not available. Funding issues have not yet been decided on by the EB. We are hoping for the best regarding funding for the possible Dagstuhl Sprint.

(2) Approval/revision of Sprint deliverables and Overview document

Flavio has provided some comments to the overview document. Arofan will update this and send around for approval by the Sprint participants.

Hilde has provided comments to some of the task related documents. Arofan will update the draft documents according to any incoming comments, sign off and prepare the draft for publications as Sprint deliverables.

(3) Status of Production framework

Oliver is very close to have an xsd.psm ready from the output from the Sprint. He hope next week to have a stable canonical xmi based production line ready for production.

(4) Organizing post-Sprint work

A. Modelling issues: 

    (a) Finalizing the guidelines

Flavio and Achim will start next week to work on the modeling guidelines document, based on the DDI4 Core UML Modeling Recommendations document developed at the NADDI Sprint.

After the Core production model can be updated accordingly.

    (b) Moving forward with Collection Patterns

The task group together with Flavio and Dan G. will look into the draft they produced in relation to examples. Larry will provide examples. Hilde could also provide an example if relevant.

    (c) Refining the production model from the Lion pull

After the UML modeling guidelines are final the work to update the Core production model can start.

B. Data description work: 

Jay will prepare a high-level  description for Process for next meeting.

    (a) Completing the examples document; 

Hilde will follow up on this in collaboration with people from the team. The documentation document from the NADDI Sprint will be updated with documentation of new data types and adding snippets from the model. This is an ongoing task.

    (b) Multi-dimensional model; 

Dan G. has provided a proposal based on data from the BLS. Discussions regarding how best to represent cube/multidimensional data have been ongoing by emails to the srg list while the week. The Data Description group with addition of Arofan and Flavio will meet on Wednesday before the MRT meeting to discuss further. The goal is to provide a proposal that can be presented to the larger group later on.

    (c) non-SQL (etc.) data; 

This is a new sub-task. Flavio and Jay are meeting regularly for discussions regarding the representation of different data types. They will get back to the larger group with proposals on a regular basis.

    (d) Event data model

The proposal from the NADDI Sprint is close to final. The proposal will be reviewed by the same group as for b) before presenting to the larger group. Jay will prioritise to review this to other tasks.

Other

Hilde has added sub-tasks in Jira for B. a, b, c and d post-meeting.

DDI4 MRT Virtual meeting 2019-05-01

Agenda (as in invite of 2019-05-01)

(1) Review of Sprint documents by participants

(2) Updating Jira issues (Hilde)

(3) Organizing MRT work moving ahead

(4) Production/Tools update

Minutes DDI4 MRT Virtual meeting 2019-04-30

Attendees:  Achim, Arofan, Dan G., Flavio, Hilde, Jay, Wendy

Apologies:  Jon, Oliver

(1) Review of Sprint documents by participants

Arofan asked the participants about reactions to content and presentation of  the latest version of the task related documents from the Sprint, that have been edited afterwards for review by the Sprint participants:

2_MRTScope.docx

3_Modelling recommendations.docx

 4a_Documenting Data Structures.docx

4b_ClassesForTall Layouts.docx

4c_Describing a Data Cube in DDI4.docx

5_FunctionalViews.docx

6_RelationshipToOtherStandards.docx

7_Annotation-Access-Provenance.docx

Flavio pointed out that the proposals included in the Modelling recommendation document were agreed at the NADDI Sprint.

Status: Agreed

Arofan asked everybody to look through the documents and get back with reactions to them as Sprint deliverables. For 4a work remains to link of each described structure to snippets of the model.

(2) Updating Jira issues (Hilde)

Hilde has started to look into the DMT Jira issues and how they are related to ongoing MRT tasks. Issues will be linked to related tasks.

An MRT task should not be closed before related issues have been closed or moved from the task.

Hilde will liaise with Arofan regarding how DMT issues should be approached. She will report back to the group regarding the handling of DMT issues in Jira.

(3) Organizing MRT work moving ahead

At the end of the Sprint a draft plan for MRT work in the forthcoming time was outlined, and the following short-term tasks were agreed:

Achim and Flavio, task starting from Starting the week of May 13^th:

• Update the DDI4 Core according to the guidelines agreed at the NADDI Sprint
• Write a modeling guidelines document.

Jay, task to start:

• Data description: Create understandable examples for noSQL data and keyValue pairs.
• Describe the Process model – what is it and how can it be use. Possibly with examples from the Alpha network and StatCan?

Hilde and Arofan, ongoing task:

• Look into DMT issues in Jira and link to related MRT task.

Oliver, ongoing task:

• Update production framework

Achim, ongoing task:

• Transformations from tool specific xmi to Canonical xmi.

 Further tasks to adress:

• Representations (bindings) of the Core needs to be created (RDF: OWL; xml:xsd) .
• Discuss Views, Alignment with other standards, Annotations and citations
• Proposal for a new task:

Flavio proposed that the modeling of multidimensional data could be made more generic than just to represent a cube (cell by cell rather than dimension by dimension). Dan G. will provide examples from the BLS.

Work to will be ongoing to organize the further work in terms of which tasks should be addressed before, while and after a possible Dagstuhl Sprint meeting.

(4) Production/Tools update

As the International Worker’s day 1^st of May is a public holiday in Europe, Oliver was not available at the meeting.

Arofan will contact Oliver in order to get an update of the production framework at the next meeting.

DDI4 MRT Virtual meeting 2019-04-17

Agenda (as in invite of 2019-04-16)

(1) Review NADDI Sprint agenda

(2) DDI Core (What do we pull from Lion? Who? How? And what exactly is the "core" anyway?, etc., etc.)

(3) Status Update on Production Framework/Editing (Oliver/Achim)

(4) Status update on Data Description Examples - identify Sprint process and goals

(5) Status update on Collections Pattern and  Modelling Issues - identify Sprint process and goals

Minutes DDI4 MRT Virtual meeting 2019-04-17

Attendees:  Achim, Arofan, Dan G., Flavio, Hilde, Larry, Oliver, Wendy

Apologies:  Jay, Jon

(1) Review NADDI Sprint agenda

Achim stressed the importance to deliver at the Sprint in order to prevent future meetings to be put at risk.

Arofan referred to the Sprint agenda (v2) and asked if something important things are missing.

Identified tasks as of v.2 of the NADDI Sprint agenda is:

• Core Modelling Issues
• Example Documentation: Application of Datum-Based Approach to Different Structures
• Functional Views/Sub-Setting of the Model
• Alignment and Use of Other Standards/Round-Trippability of Bindings

(Functional Views/Sub-Setting do not need to be complete, and Alignment and Use of Standards/Round-Trippability of Bindings is on the wish list).

(2) DDI Core (What do we pull from Lion? Who? How? And what exactly is the "core" anyway?, etc., etc.)

A topic that was brought forward is to get the DDI Core framed out. Achim pointed out that a common understanding of what the DDI4 core comprises is important.

On a high level Conceptual, DataDescription and Process has been mentioned as describing the DDI4 Core. Arofan suggests that a goal for the meeting could be to create a more detailed ‘base’ list of what DDI4 Core comprises. Inclusion criteria need to be defined. Flavio points out that some usecases that show why the DDI Core is important could be useful.

               Status: Agreed to add DDI Core Scope as an additional topic of the NADDI Sprint agenda.

(3) Status Update on Production Framework/Editing (Oliver/Achim)

Oliver will prepare pipelining before the Sprint so that people can do things on their own machines. This will consist in plug-and-play or instructions.

Oliver will be able to attend the meeting virtually in the mornings of the 23^rd and 24^th Ottawa time.

(4) Status update on Data Description Examples - identify Sprint process and goals

Dan G. reports that specific kinds of tall/skinny structures remain to be worked out.

Not all the created documentation is, however, easily accessible.

Hilde referred to Jay’s work (powerpoint attached to MTR task DMT-234) as an example of understandable documentation. Jay examples are, however, based on some remodeling.

Arofan suggests that a goal for the meeting could be to create a ‘dummies guide’ to the model as it is, using a similar style as in Jay’s work and the Variable Cascade presentation from the last Dagstuhl workshop, and identify gaps in the current model.

Arofan will provide an outline of the deliverable before the Sprint.

               Status: Agreed

(5) Status update on Collections Pattern and  Modelling Issues - identify Sprint process and goals

On the meeting of April 2019-04-16 the CollectionPattern was discussed. DMT-232 Patterns and Collection Task group (Larry, Jay, Achim) has provided a draft revision of the Collection Pattern called ‘CollectionPatternDiagramPerApplication’, sent by Larry to the srg list 2019.04.10. Flavio provided on the meeting day a draft revision of the Task group’s Collection Pattern proposal and sent out an update of his proposal after the meeting. The draft provided by the Task group is already pretty mature. Some discussion of the draft based on the input of Flavio is needed.

For the Sprint Achim and Wendy point out the need for a detailed and consistent set of modelling rules. Existing guidelines need to be discussed and details need to be added. Larry and Flavio point out that how to apply things is also important: E.g. Realisation vs. Inheritance for patterns, Interface vs. Implementation/bindings.

Existing modelling guidelines, the ‘UML to be Decided’ document from the Berlin Sprint and Post-prototype issues (to be provided by Wendy before the Sprint) could serve as basis for this work.

               Status: Agreed

Other

Round-Trippability of bindings is only the wish list for the Sprint, but it could make sense to discuss what Round-Trippability means to frame the discussion and to consider some general principles.

 A draft of what a View is should also be made.

Arofan will follow up on the work at the Sprint and will edit and compile the Sprint report.

To do before the Sprint

• Arofan to provide an update of the agende and Hilde to publish
• Oliver to provide facilities for pipelining
• Wendy to provide post-prototype issues
• Arofan to draft the Data Description task for the Sprint

DDI4 MRT Virtual meeting 2019-04-03

Agenda (as in invite of 2019-04-01)

(1) Sprint-related items: update (if needed) on NADDI sprint, and discussion of possible other sprints at EDDI, elsewhere (Achim)

(2) Update on production system/COGS (Oliver)

(3) Status update - Modelling issues

(4) Status update (Data Description examples)

(5) Testing/Binding issues (time permitting)

(Other: Guidelines to JIRA (Hilde))

Minutes DDI4 MRT Virtual meeting 2019-04-03

Attendees:  Achim, Arofan, Dan G., Flavio, Hilde, Jay, Jon, Oliver, Wendy

Apologies:  -

(1) Sprint-related items: update (if needed) on NADDI sprint, and discussion of possible other sprints at EDDI, elsewhere (Achim)

NADDI Sprint: Hilde has put up a Sprint page for the Ottawa Sprint. Related pages will be filled with content before, during and after the Sprint (pictures from Ottawa kindly provided by Flavio).

Alternatives for future Sprints were discussed. Achim proposed two alternatives for a second Sprint meeting this year:

1) A sprint meeting held at Dagstuhl in the week of 30^th September through 4^th October.

2) A sprint meeting in connection with EDDI in Tampere, either before from 28^th through 29^th November, or after 5^th through 6^th December.

The Dagstuhl alternative was chosen by the group. It was decided to make a proposal for the AG that a Sprint meeting is organised for the week 30^th September through 4^th October in Dagstuhl.

All MRT group members are interested to take part in the possible Dagstuhl Sprint.

Availability of people (as of April 3^rd) is indicated below:

• Achim is available, organising
• Arofan, Jay, Larry and Wendy are available, need funding from the DDI Alliance
• Flavio and Hilde need approval from their institutions
• Jon and Dan G. are possibly available, funding not needed
• Oliver is available, funding not needed

Topics for the Dagstuhl Sprint will be discussed at the Ottawa Sprint and after. General topics are fine-tuning of DDI4 Core regarding what to include and what not.

In addition to the Dagstuhl Sprint, a proposal for a Sprint meeting in connection with next year’s NADDI will be proposed.

                Status: Agreed

(2) Update on production system/COGS (Oliver)

This section of the meeting was divides in two:

1) Update

2) We need a place to work – where/how work in the short term

1) Update: Oliver has been working on transformation canonical XMI to COGS and tested it. The test is not 100% complete but will be final in the short term (perhaps for the next week). XMI export from COGS is possible, but this is not Canonical XMI. A question is how identification should be handled.

A goal is to have Canonical XMI back and forth from COGS. It is currently difficult to set a timeframe for the completion of the work.

The pipeline in BickBucket will be used to run the COGS info and the back to COGS transformation, as described in the document developed by Oliver with input from Achim, sent by Achim to the srg list 2019-03-27.

                Status: Agreement to get the production process up and running as soon as possible.

2) Where/how to work

Achim suggests that we need to distinguish between local changes to the model (documentation, multiplicity etc.) and bigger structural changes.

For the time being we can document local changes as JIRA issues.

For the bigger, structural changes people can work in their own systems. Agreed solutions can be provided Flavio for implementation in EA. This has an XMI output that can be transformed to Canonical XMI (Achim has been working on this and will talk about it in a meeting soon). The outcome can then be stored in the repository that Oliver describes in a controlled way.

                Status: Agreed

For the next time

• Larry to present the new proposal for the CollectionPattern ‘Collection pattern version 2019-04-03’, as sent by Achim to the srg list on April 4^th.
• Arofan and Achim to prepare material on bindings
• Larry to update on data description examples – wide and tall data structure.

Other

Hilde has put up and sent around a short guide to the entering of MRT content in JIRA. The guide will be published on the MRT Task pages in Confluence.

DDI4 MRT Virtual meeting 2019-03-27

Agenda (as in invite of 2019-03-26)

(1) NADDI Sprint Update

(2) Modeling issues (as raised in Flavio's e-mail, others)

(3) Report from Data Model Examples team

(4) Production Lifecycle (Achim/Oliver)

(5) Test Plan/Testers

Minutes DDI4 MRT Virtual meeting 2019-03-27

Attendees:  Achim, Arofan, Dan G., Flavio, Hilde, Jay, Oliver, Wendy

Apologies:  Jon

1. NADDI Sprint (any news?)

Achim reports that the EB has received the agenda for the Sprint. No comment has been received on this.

2. Modeling issues (as raised in Flavio's e-mail of 2019-03-25, others)

Flavio points out that a requirement for having patterns is to capture commonalities across seemingly disparate classes to be used to create pattern-specific binding rules. So instead of having only generic rules that represent all relationships between classes in the same manner across the entire model we could have tailored representations for classes and relationships that were part of a pattern.

Achim points out that requirements for having patterns will need to be looked further into, and transformations for bindings need to be consolidated. More examples regarding usage of patterns are needed.

Different types of documentation would be needed for different usages/perspectives. Not all would require the abstract patterns per se. Patterns are e.g. good for developers but not so good to understand the model.

Although the Canonical XMI always must represent the full model, there is consensus in the group that different documentation is needed for different user-groups/use-cases. Flavio brought in the concept of Persona: A person by task or with a different hat.

Achim describes the perspective of an overall model, views as a sub-setting of the overall model, and persona that are usages for a specific function.

Jay, Achim and Wendy suggest that we could try to auto-generate documentation for views and personas (viewpoints) if they can be formalised.

There is an ISO standard for software architecture (link provided by Achim after the meeting) that could be useful to look into regarding persona. Arofan will provide a draft of persona for the group to comment on. These can be compared to ISO and where they correspond ISO could be referenced.

3. Report from Data Model Examples team

Dan reports that the group is making progress and will provide something to discuss next week.

4. Production Lifecycle (Achim/Oliver)

Proposal for production framework by Oliver with input from Achim will be circulated before next meeting.

Topics for agenda of the next meeting

• Look into the collection pattern to prepare for testing on representations based on Flavio’s idea.
• Production framework: Should work be done in Lion before a different framework is in place? Canonixal xmi could be used. Short and long run perspective.
• Testing plan/testers.

To do for the next time

• Arofan to write up a document about Personas (see emails from Arofan and Wendy to srg list of 2019-03-27).
• Achim to provide info about ISO standard for software architecture (see email from Achim to srg list of 2019-03-27).
• Achim and Oliver to provide document about production framework including proposal for revision (see email from Achim to srg list of 2019-03-27.
• Hilde provide guide to task management in Jira.

DDI4 MRT Virtual meeting 2019-03-20

Agenda (as in invite of 2019-03-19)

1. NADDI Sprint (any news?)
2. Modelling Issues report (Larry to present)
3. UML Modeling tools (Achim to present)
4. Data Description Examples status update
5. Production Framework status update
6. Testers/Testing Process

Minutes DDI4 MRT Virtual meeting 2019-01-30

Attendees:  Achim, Arofan, Dan G., Flavio, Hilde, Oliver, Wendy

Apologies:  Jay, Jon

1. NADDI Sprint (any news?)

Achim reports about positive reactions to the draft NADDI Sprint agenda from the recent AG meeting. The agenda will be discussed at the next EB meeting (today, 03-20)

Based on the document ‘DDI Roadmap v2.0’ sent to the srg list 2019-03-20, the group decided to add a point to the NADDI Sprint agenda that regards the relationship between the MRT work and relationships to earlier versions of the standard.

                Status: Agreed

2. Modelling Issues report (Larry to present)

The outcome of the work by the group as described in the related document (of today 03-20) was presented by Larry.

Patterns

• Cluster analyses: Different approaches of cluster analyses of Class to Class, Package to Package and View to View has been performed. Results could give an indication of how things could be grouped together (applies to Packages rather than Views).
• Multiple inheritance: The ‘diamond problem’ was explained in relation to multiple inheritance. Most of the multiple inheritances in the model are due to Identifiable and Annotated Identifiable.
• One possibility regarding identification is to make identifiable a property (see related document). If identification is modelled this way only 4 instances of multiple inheritance will remain.
• Another proposal is to move associations out of structured data types. These could be classes or remodeled.
• Views could be expressed as aggregation of classes (aggregation can be understood by all UML tools).                            

Collections

                See proposal for changes in the related document.

Status modelling issues: Further discussion required. Flavio will provide comments to the proposal for changes to the model.

3. UML Modeling tools (Achim to present)

Achim has been working on the DDI4 Canonical XMI transformations from a variety of UML tools. This to allow for an interoperable model not bound to a specific tool.

UML 2.4.1 and later has been in main focus.

Findings are that all examined tools have a custom XMI export flavor that can be transferred to Canonical XMI.

Visio Professional has not yet been tested.

Status: Dan G. will provide an export from Visio professional for Achim to test.

Achim will have a sub-set (Canonical XMI xslt stylesheet) available for testing by the group for the next two weeks. The task can be finalized after this.

See related documents from Achim:

DDI4 and Canonical XMI

DDI4 Object Inventory

Items for transformations

5. Production Framework status update

Oliver and Achim have been discussing Canonical XMI in connection with the production framework. A new solution is on the table that represents some adjustments to the diagram presented by Oliver at the last meeting. This involves consumption and export of Canonical XMI for the production framework. Interactive work could be done in modelling tools.

Oliver points out that a consequence of the new approach will be that a pipeline will not start automatically by each push, but be performed on a regular basis by certain people.

                Status:  Agreement on the idea. Oliver in collaboration with Achim will provide an update of the proposed solution for the group and other relevant parties.

4. Data Description Examples status update

Dan G. reported from the work of the Data Description group. The group meets every week and will be looking into mainly three data structures:

• Short/fat (rectangular data files)
• Tall/skinny (event data)
• Cube (dimensional data/aggregate).

Documents will be provided for group discussion as the work proceeds

               Status: Present something at the meeting next week.

Flavio asked about the relationship to GSIM. Dan G. pointed out that GSIM is a conceptual model while DDI is an implementation model with reoresentations. The Data Description is very close in GSIM and DDI, and GSIM is currently taking up DDI features like the differentiation between substantive and sentinel values and other.

For the next meeting:

• Process and Collections: Flavio comment on Larry’s document.
• Production framework: Oliver in collaboration with Achim, prepare revised proposal regarding the production framework and send to the group.
• Dan G. provide something for the group regarding Data Description.

DDI4 MRT Virtual meeting 2019-03-13

Agenda (as of invite of 2019-03-12)

(1) Update on NADDI sprint (I am hoping this is now very quick)

(2) Jira/Confluence presentation from Hilde

(3) Status update from Modelling Issues group (if they are ready to do so)

(4) Status update on Datum Application examples work

(5) Status update on Modelling tools

(6) Status update on Production Tools

(7) Discussion of testers/test process

Minutes DDI4 MRT Virtual meeting 2019-03-13

Attendees: Achim, Arofan, Flavio, Hilde, Jay, Larry, Oliver, Wendy

Apologies: Dan G., Jon

1) Update on NADDI sprint

NADDI Sprint agenda: Achim has made slight modifications to Arofan’s draft of the agenda. This is now ready for discussion at the AB meeting on the same day. The agenda (with possible updates) will be sent to the EB for their next meeting.

Status: Agreed

2) Jira/Confluence presentation from Hilde

Hilde presented shortly Confluence pages under development for the MRT group.

Thereafter focus was then put on demonstrating possibilities for creating and maintaining tasks and sub-tasks in Jira, as well as how these can usefully be displayed on the MRT tasks pages in Confluence.

The proposal for handling of MRT tasks in Jira and Confluence is included in the document ‘MRT tasks in Jira and Confluence’ sent by Arofan to the srg list with the meeting invite of 2019-03-06, as well as in the Appendix.

                Status: Approach to develop and maintain MRT tasks in Jira with links to MRT task pages in Confluence agreed.

Hilde will write up a short guide on how to enter MRT tasks in Jira. The proposal will be reviewed by Wendy and Achim.

Hilde was asked by Arofan to coordinate the MRT tasks and related issues in Jira and agreed to that. Wendy will make Hilde owner of the related Jira pages.

                Status: Agreed

3) Status update from Modelling Issues group

Larry and Jay have been focusing on design/modeling principles, trying to simplify the model where possible. Focus is put to preserve reusability but to reduce the use of explicit associations in the UML.

Jay and Achim have been looking into how Views could be restructured taking into account different types of data structures. Achim has been working on options for creating Views using UML.

Cluster analyses have been performed for the purpose of looking at relationships between classes and packages. Classes in the model have been analysed according to how many times they have been referenced.

The group agreed to present their work at the next meeting.

Status: Agreed

4) Status update on Datum Application examples work

The Data Description group has met once. The next meeting is on the forthcoming Monday.

Achim reports that the group is looking at a list of structured data types when it comes to what is currently possible in the model and what should be while this year. The group has decided to look at the model both bottom-up and top-down.

The bottom-up approach will focus on how to describe data structures using examples. Achim and Larry are starting this. Hilde will join in on this task. Jay and Dan G. will handle the top-down. Things that are new in DDI4 should be kept in focus.

Arofan asked the group to present some results of their work at the next meeting and they agreed to that.

6) Status update on Production Tools

Oliver reported from work related to production tools. A note describing his proposal related to the production tools is included in the Appendix.

A question regarding Atlassian license used and capacity needed was brought up. Oliver will check out regarding license possibilities (see Appendix).

Status: To be discussed further

For the next time:

• Larry and Jay will report back to the group from the modelling issues task on process and collections.
• The Data Description group will report back from their work
• Achim will report back regarding UML tools
• Testers and test processes will be a topic at the next meeting.

Appendix

MRT tasks in Jira and Confluence, proposal Hilde

Structure

• MRT tasks can be added hierarchically as Tasks and Sub-tasks in Jira. An example of a task with a sub-task can be found here.
• Each task or sub-task can be linked to related issues to be resolved.

Relationships in Jira projects are reciprocal, so if an issue is linked to another issue or task the relationship will show up in both issues.  

• MRT tasks can have working documents in progress related to the tasks attached to them. A General modelling issue task could for example have the UML To Be Decided – Consolidated google doc attached to it.
• An overview MRT task page has been developed in Confluence as a sub-page of the MRT-pages. This page includes a filter linking up to Jira tasks kindly provided by Wendy. Any task or sub-task labelled ‘MRT-Task’ will currently show up on the confluence page if its status is ‘Open’, ‘To do’, ‘In progress’.

Workflow

• A group of people from within the MRT coordinating team is assigned a work task. A contact person is appointed for the task.
• A task or a sub-task labelled ‘MRT-Task’ is developed for it in Jira. The task will then show up in the MRT tasks overview in Confluence.
• Related issues, as well as related working documents (google docs) are linked to the task and updated as the work is progressing.
• Proposals for solving the task are discussed with the full group at specific milestones. This can be in-group virtual discussion meetings, face-to-face Sprint meetings, discussion with external experts etc. For each milestone a stable pdf or word version of the document with a specific version number is produced and published at the MRT milestones page in Confluence for agreement.
• When a decision regarding the task has been reached the discussion document is updated, and a new stable version that includes the decisions of the full team is produced and published at the MRT milestones page.
• The agreed solution is implemented in the MRT production process and moves forward to iterative MRT testing. The status of the task can be changed to ‘done’ when it is consider completed and should not be visible in Confluence any more.

MRT Sprint pages’ relationship to MRT tasks and milestones pages

The NADDI Sprint pages could contain rather basic information like venue, programme, attendees, focus on the sprint (tasks to focus on etc.). Then a links to the MRT task page and MRT milestones pages could be provided, where information about the tasks to take on can be found.

Production framework proposal, Oliver

The future production platform for DDI 4 will probably be based on BitBucket Pipelines. Assuming, that the DDI Alliance has a regular Attlassian academic subscription, we will have 500 min/month to run on the Attlassian servers. We will have an eye on the runtime of the current steps to figure out, if we run into billing with this.

The suggestion for the general setup looks like this:

A first repository will contain the necessary files in COGS format. This Repository will be equipped with a pipeline to run the COGS task for creating XMI. This task will be adopted so that it will be compliant with canonical XMI. The result will be pushed to the second repository.

The second repository will be the current ddi-views-production repo. In this repository I already included the current canonical XMI. The pipeline in this repository will run the current XSL transformations for the binding and documentation production. A second pipeline will contain the RDF-binding production. The result will be pushed to the third repository.

The last repository will be based on the current ddi-views repo. Besides the result of the pipeline it will contain the manually written documentation parts. The further production will be the HTML and PDF production based on Sphinx. The runtime of this step has to be evaluated concerning runtime.

Besides that, the second repo could be used for the integration of UML tools. The first step is to pull that repository to get access to the canonical XMI source. After changing the UML in the tool, the content gets exported as XMI (in canonical or “proprietary” form). The repo will also contain a set of transformations and the according execution scripts for each UML tool to create the related COGS files.

DDI4 MRT Virtual meeting 2019-03-06

Agenda (as of invite of 2019-03-05)

1. NADDI Planning, including review of draft for EB/Community
2. Status updates

    - Modeling issues (Larry/Jay/Achim)

    - UML Tools

    - Production Tools

1. Review of Hilde's Jira/Confluence proposal (Please see document, attached)
2. Test process and candidates

Minutes DDI4 MRT Virtual meeting 2019-03-06

Attendees: Achim, Arofan, Dan G., Flavio, Hilde, Jay, Larry, Oliver, Wendy

Apologies: Jon

1. NADDI Planning, including review of draft for EB/Community

Arofan’s document ‘NADDI Sprint Draft Agenda’ sent in email to the srg list 2019-02-27 was discussed. There was agreement to adjust wordings regarding Sprint deliverables. It is important to make clear that (at least) the following three tasks mentioned in the document will be worked on at the Sprint:

• Core Modelling Issues
• Example Documentation: Application of Datum-Based Approach to Different Structures
• Functional Views/Sub-Setting of the Model

A report for each will form basis for the Sprint deliverable report. A goal for the Sprint will be to detect issues. This will be reflected in the Sprint report. It is not expected that the three work tasks will be completed at the Sprint, but the Sprint will represent a milestone for the work in progress.

• The Core Modelling document for the task will for example not be finalized at the Sprint, as Core Modeling work will be needed afterwards.
• The Examples Documentation of the Datum-based Approach might be completed for some, but not all data structures.
• The outcome of the Sprint when it comes to Views is expected to be a collection of ideas and thoughts, rather than a resolution.

NADDI Sprint Roles and organization:

• Arofan was asked by Achim and agreed to chair the Sprint meeting.
• Hilde was asked by Achim and agreed to help with minutes as well as coordinating documentation and issues in Jira and Confluence.

Achim will adjust Arofan’s draft agenda for the NADDI Sprint to reflect the above thoughts.

                Status: Agreed

2. Status updates/ Data Structures discussion

- Modelling issues: Achim reports that work to analyze components of the DDI4 Core model to detect issues is ongoing. Larry, Jay and Achim is working on this.

- Document on Data Structures: Achim has started to draft the document on Data Structures, see attachment in email from Achim to srg list of 2019-03-05. Larry, Dan and Jay will meet a couple of times before the NADDI Sprint to develop the document further. Examples will  be provided to make sure that the LogicalDataDescription part of the model can be used in common cases for typical data structures, including aggregate data. Jay will provide examples from the Alpha Network. It is a goal for the report from the Sprint to relate to common cases. The document will represent a first step to model revision.

If/when the tall/skinny data structure discussion would add inn would need further discussion.

Dan G. asked if we need to make a plan for solving identified gaps. Arofan points out that the planning of the Sprint is the first step in this direction.

Arofan asked Larry, Jay, Dan G. , Achim and Flavio to contribute to the documentation of data structures work.

• Larry, Jay, Dan G. will work on the task.
• Dan G. agreed to be the contact for the task.
• Achim will help to structure the document.
• Flavio will have the role as a reviewer.
• Arofan mentioned that Jon could be asked to contribute as an additional reviewer.
• Hilde agreed to contact Ørnulf regarding possible review contribution.

                Status: Agreed

- Production Tools framewok: Oliver follows up on this.

- UML tools: Achim is currently exploring xmi flavors of different UML tools (first and foremost Eclipse and Enterprise Architect) and their transformation to canonical xmi. He is liaising with members of the group regarding testing. Flavio will do Enterprise Architect tests. Visio-pro could also usefully be tested etc..

Tasks for next time:

• Achim revises the ‘NADDI Sprint Draft Agenda’ document.
• Work on ongoing tasks continues.

DDI4 MRT Virtual meeting 2019-02-27

Agenda (as in invite of  2019-02-26)

Goal: To make necessary plans for the NADDI Sprint (as appropriate) and to start working on the data-related modeling tasks

(1) NADDI Sprint update and planning (Achim)

(2) Update from modelling tasks (Jay/Larry)

(3) Update on modelling tools (Achim)

(4) New task on documenting data structures (as discussed earlier)?

For Next Week:

(5) Testers and Testing Process

(6) Confluence/Jira for Tasks and Issues (Hilde)

Minutes DDI4 MRT Virtual meeting 2019-02-27

Attendees:  Achim, Arofan, Dan G, Flavio, Hilde, Jay, Larry, Wendy

Apologies: Jon, Oliver

Pre-agenda topic: Meeting times in March.

March is transition month for DST at different dates on different continents. We agreed to follow the TC line regarding meeting time provided by Wendy: North America/Ottawa will shift on Sunday, March 10. Europe will change on March 31. So for the meeting dates 13^th, 20^th and 27^th March, some members will have already "sprung forward" and hour. As we set our schedule by Central European Time (whether DST is in force or not) this means that for the meetings above North Americans and Canadians will meet an hour later.

1) NADDI Sprint update and planning (Achim)

Funding for the Sprint was approved on two requirements:

(1) incorporate DDI 4 prototype review feedback into the sprint, and

(2) submit back to the Executive Board concrete deliverables that will be produced by the end of the sprint. 

Practical planning of the NADDI Sprint

Flavio helps with the practical planning.

• Meeting rooms: A meeting room has been booked in a hotel for the Monday (cost about 200$, covered by the DDI Alliance funding, possibly others). For the Tuesday and Wednesday rooms at StatCan are reserved.

  Note:  Inner security rules at StatCan imply that people will be taken around by Favio or other StatCan staff when moving around inside the building. People can bring their own laptops.

• Lunch/food for breaks: There is coffee in the hotel, Flavio will ask for additional snacks. At StatCan the cafeteria can be used for lunch. People pay for themselves.

  Flavio will ask regarding additional coffee, snacks etc. for the meeting breaks.

• Hotel rooms: 10 Individual hotel rooms have been booked. There is a special price near NADDI. People need to order rooms themselves. Flavio will check what the booking keyword will be.

  Participants keep their receipts and send them to the DDI Alliance to get refunds the usual way.

Sprint Content

The two requirements of the EB, which both make sense, should be in focus regarding the sprint. Achim will look into results from the prototype review prior to the meeting.

Aims for today's meeting: 1) to develop a realistic agenda for the meeting: What could usefully be achieved at a face-to-face meeting, and 2) a pre-NADDI agenda for the next meetings.

Achim suggests that complex issues that require an iterative approach and discussions back and forth would be useful to put on the agenda for the face-to-face meeting.

Examples are design patterns and collections that Jay, Larry and Achim currently are looking   into. How to simplify the model and reduce the amount of classes and relationships?

Documentation of datum-based model application to examples is a different topic.

 A third topic for the Sprint suggested by Achim is how to subset the library or make views. It is important to be able to create a sub-set of the model for a particular purpose and to make the model simpler and less connected.

Arofan indicated additionally that another possible task for the meeting could be standards alignment and roundtripping.

Agreed plan for the NADDI Sprint:

• Core modelling issues (all)
• Data structures examples and modelling (breakout group)
• Views and sub-setting (breakout-group)
• Standards alignment (not sure if there will be time for this)

The work process would be aiming at developing statement or draft of statement documents including alternatives, discussions and conclusions on each of the identified topics of the agenda (high level tasks) it should be related to and syncronised with the identified Jira issues for each topic/task. Issues from the prototype review will be highlighted.

The individual documents will form basis for the complete report for the EB, but also to serve as documentation of the MRT for the group itself and others. Issues from the prototype review will be highlighted.

To do/pre-NADDI preparations

•  Arofan will draft a work plan for the NADDI Sprint and to send to the group for comments.

To start the process a set of topic/task related documents will be drafted:

• Achim will start on a draft topic document for documentation of datum-based data structures. This topic needs to be described in an accessible way so that members of the community can understand it.

 The Variable Cascade presentation from Dagstuhl provides an example of good documentation of conceptual issues. Goals for the final document: 1) Describe the effect of findings on the model. 2) Provide good documentation.

           Achim suggests to create a list of people not at the meeting to look into this. Ørnulf and Jon/Darren were mentioned. Hilde will contact Ørnulf regarding possible contribution on the topic.

• Flavio will start on a draft topic document on Views.
• Larry, Jay and Flavio were asked to start to draft a document about standards alignment. This task has however the lowest priority.

The above documents will be further developed, and hopefully agreed while the face-to-face meeting.

• Other pre-NADDI topics would be to review modelling guidelines, as well as to collect filed issues belonging to specific high level topics or tasks. Work regarding the latter started while the Berlin Sprint. Larry and Jay have been taking up on this.
• Wendy will review the status and triage of issues and will get back to the MRT group, first mid March and again closer to the meeting.  Achim will look into results from the prototype review.

For the next meeting:

• Take-up from this meeting
• Talk briefly about roles in the MRT group and virtual meetings, and at the Sprint.

DDI4 MRT Virtual meeting 2019-02-20

Agenda (as in invite of 2019-02-19)

Goal: To agree a plan for identifying the short-term production framework, and to determine status on other tasks/infrastructure

(1)  Update on proposed NADDI Sprint (Achim)

(2) Hilde to put up a draft Tasks page (Hilde)

(3) Update on Modeling task (Jay/Larry)

(4) Production framework: identify alternatives and get people to draft proposed solutions (if more than one) for short-term and long-term

Minutes DDI4 MRT Virtual meeting 2019-02-20

Attendees:  Achim,  Arofan, Dan G, Flavio, Hilde, Jay, Larry, Oliver, Wendy

Apologies:   Jon

(1)  Update on proposed NADDI Sprint

Achim reported that the EB would meet today (20^th February). If there are some news regarding the possible NADDI Sprint he will get back to the group with them.

(2) Hilde to put up a draft Tasks page

Hilde has created a basic MRT page with some sub-pages. She will get back to the group regarding what a task page can look like and how Jira issues could be linked to the page. Wendy will assist regarding how to use Jira.

(3) Update on Modeling task

This is ongoing work related to collections and design patterns in general. Larry and Jay report that they are producing a document with alternatives, pros and cons on which the group can make decisions.

Some examples of what they have been looking at:

• Collections: Relationships are not always shown in the diagrams. Proposals on how to solve this is in progress.
• Separation of semantics and structure. For example group type things in DDI3. DDI4 has a smaller set of classes. Solve semantics differently than having this and that kind of group. A general goal is to reduce the amount of classes in the model.

Achim brought up the topic of how to use UML with collections design patterns. There are some differences between how design patterns are handled by different versions of the UML. Achim will assist Jay and Larry with this and will take part in their calls regarding the task.

(4) Production framework

Achim explained the production platform currently imports exports from Lion. This is not Canonical XMI. The import of the production platform needs to be changed to Canonical XMI. Additionally, transformation to OWL from Canonical XMI needs to be integrated in the production framework. This requires a Javascript environment.

                Status: Agreed

Tasks

Arofan summarises tasks as follows:

1. a) Change import to the production framework from Lion to canonical XMI.
2. b) Look into canonical XMI export from UML tools (Achim is already looking into this).
3. c) Integrate OWL in the batch workflow (see above).
4. d) Not for the immediate term: Look into how canonical XMI works with COGS. (Achim agreed already to look into model-to-model transformations in the Eclipse tools).

Oliver volunteered to do a) and c) and to look into d). Work on d) is in progress. Achim’s hint to Oliver: IDs in Canonical XML are regular XML IDs.

Achim will continue with his tasks and agreed to contact Eric regarding RDF and which components should be run in which pipelines.

Jay and Larry continue with their modelling tasks with help of Achim.

Other

Jay proposed that we could start to work with representation even if this is not ideal. Alternatives for this will be followed up on in the upcoming meeting.

DDI4 MRT Virtual meeting 2019-02-13

Agenda (as in invite of  2019-02-12)

Goal: To identify a course of action for modelling tools and launch immediate modelling tasks

Agenda:

(1) Modeling tool proposals

(2) Organizing short-term modeling tasks

(3) Production framework

Minutes DDI4 MRT Virtual meeting 2019-02-13

Attendees:  Arofan, Dan G, Flavio, Hilde, Jay, Jon, Larry, Oliver, Wendy

Apologies: Achim

(1) Modeling tool proposals

Arofan referred to the discussion between the meetings (included in the Appendix).

Some further investigation is required regarding the modelling tools. Achim will look into requirements and the possibilities for Canonical XMI export (see email from Achim to srg list 2019-02-08 in the Appendix).

Oliver’s layout (see email from Oliver to srg list 2019-02-08 in the Appendix) could work as a practical solution for the next steps.

            Status: Agreed

Control of the single source of truth

Jon brought up the issue of version control and pointed out that a good proposal on the triangulation between model, Canonical XMI and user management is needed. The discussion on this has started at earlier meetings, and some requirements regarding versioning were formulated at the Berlin sprint.

The discussion was centered around the need for workflow plans, gatekeeping and a way to maintain the single source of truth.

Agreement was made that a strict gatekeeping would be necessary.

The following workflow was proposed by Arofan:

• Small groups within the MRT group work on special things regarding the model.
• The group makes a single proposal for discussion in the full group.
• Agreement from the full group is implemented in the model by an assigned gatekeeper.

Larry suggested that UML tools are used to enter the decisions and then submit to Git. Gate keeping could be done in advance as well, based on a policy, for example that well-established components of the model are not changed, that only new classes are added, or similar.

Flavio pointed at the need to find out which classes are important for all DDI4 core sub-packages.

                Status:  Needs follow-up work

Flavio was asked by Arofan to take the role of the gatekeeper and agreed to that.

Oliver was asked to look into possibilities regarding versioning system and agreed to that.

Views

The discussion was centered around what a View is, based on Arofans question: are Views purely documentary?

The statement that Views are just an adjustment of the model on the level of the diagram was tentatively agreed.

                Status:  Needs follow-up discussion

Flavio pointed out that if Views are to be regarded as an adjustment of the model on the level of the diagram, then the hiding of classes, associations and attributes would be required features for the chosen UML tool.

Rules for the representations need to be decided.

Simplification of the model

Jay proposed to look into patterns and collections to see if they are useful. This would be a way to explore possibilities for simplification of the model.

(2) Organizing short-term modeling tasks

Larry and Jay were asked by Arofan to look into patterns, and collections as an example of patterns. There was more agreement about collections and less agreement about patterns in general in the past. Larry and Jay agreed to, in the scope of weeks, get back to the full group with a document that  summarises  issues, makes a proposal for possible changes to the model and  describe, pros and cons.

Follow-up task:

Modelling tasks: Larry and Jay to look into collections and patterns.

Versioning: Oliver look into possibilities regarding versioning.

UML tools requirements and  Canonixal XMI transformations of ULM exports : Achim look into this.

Appendix

Correspondence since the last meeting regarding tools and tasks between Oliver, Achim and Flavio (most recent first):

Achim in email to srg list 2019-02-11

I think the question of modellling tools and achieving a working environment around the Canonical XMI is crucial. On this basis several people can use UML tools for exploring and validating new model approaches. If this is not possible, we are running into a similar situation where we already are with Lion. 

This task should have highest priority.

Surely can we start discussing mpdeling question in parallel until a well working solution is achieved.

Achim

Flavio in email to srg list 2019-02-08

Maintaining the platform and the canonical model is important in the medium term. However, it's very difficult to achieve soon considering the multiple constraints we have.

I think it's more urgent to start addressing some of the core issues we have raised regarding (i) fundamental modeling (patterns, design rules, etc.) and (ii) core deliverables (conceptual, representations, process, etc.). We need to go over Jira and start make decisions soon. We can use any tool we can to model both requirements and candidate solutions in UML, so that we can discuss with concrete diagrams. Merging into the main development branch is another story.

I agree we need solid gate keeping. In my view, one of the main problems in the past few years has been that lots of people started making changes to the canonical model without understanding all requirements. The "move fast and break things" approach doesn't really allowed us to move fast at all, just ended up creating all sorts of problems and making large parts of the model unusable for long periods of time. In other words, it just gave us a false sense of agility. I agree we need to update the model more carefully with some old-fashioned governance in place.

Having said that, we don't need to agree on how to do that right now, we can work on this in parallel. What we need to do for the next while, perhaps until NADDI, is to start looking at the foundational modelling issues that have been identified.

One issue that was not discussed so far, however, is the definition of views. The model is very hard to use without a flexible way of creating views that hide unnecessary complexity. Without that mechanism in place, we end up trying to get rid of complexity from the model itself, which creates all sort of other problems. The complexity of the model is there in many cases because we want to satisfy multiple, complex requirements. The model should be as simple as possible, but not simpler. With a powerful view mechanism we can allow people to see only what they need to see and hide the rest. I'm going to create a Jira issue to elaborate on this.

Flavio

Achim in email to srg list 2019-02-08

Oliver and others,

In general, I agree, the Canonical XMI in a git repository as central source would be a good way to go. The consequence would be that any UML tool which is used for modeling must have some way (like a post export transform by XSLT) to create Canonical XMI. The canonical (regarding the content) git repos should be managed only by a few people. But everybody can import the Canonical XMI into an UML tool of choice to play with it and to develop new approaches. The individual outcome could be then fed back into the group process, and can possibly – after agreement – go into the git repos with the help of a few core modelers.

This approach with the Canonical XMI would enable some independency from a specific tool. The Canonical XMI would have the role of a portable format. The import is usually not the issue but the export.

I’am looking into how the XMI export flavors of Eclipse Papyrus and EA can be transformed to Canonical XMI. Eclipse’s own model serialization is anyway in XMI, the Eclipse Ecore. I will look also into the requirements which an UML editor should be able to do, definition of classes, attributes, associations, and documentation.

The Eclipse Ecore format is used across any Eclipse modeling tools and can also be imported by some commercial UML tools like MagicDraw. In some way Eclipse modeling tools took over some area of Rational Rose which was the de facto standard for modeling (as tool and as model format. IBM has now the tool Rhapsody which is as expensive as Rose was but has not the central role anymore).

Eclipse has a component called CDO which enables the use of a shared remote model repository. This sounds attractive for collaborative editing of a model. But there are also some possible issues: a CDO repository server would be necessary, it works only with Eclipse, and I’m not sure if it makes sense that 10 people are editing a model without management of the editing process.

Regarding validation, the overall issue is that none of these tools is doing a wide variety of validation tests. Only the use of several UML tools shows possible flaws of the model.

Achim

Oliver in email to srg list 2019-02-08

Here are my ten cent on this:

We currently have a XMI representation of DDI 4. This is not canonical XMI but we know a way how to transform it into. Any binding further on is based on XMI. If we now are facing a tool setup that is able to directly interact with XMI, I see no need to work with any simplified representation for editing purpose.

Assuming the possibility to implement a transformation for EA and eclipse/papyrus outputs to canonical XMI perhaps even separated into a file per package to make it more handy, we could think about the following environment for distributed development bound to an automatic binding production:

This would mean, that we will have a the canonical XMI directly modeled with EA and/or eclipse shared and synchronized through a GIT repo.

For each binding (green box for each) a XSL transformation would be implemented (based on the existing) to do any kind of reduction and bringing everything into one file, if necessary. The following process of the production line would look the same, as it does now. Each binding gets created as one or more files, which might be processed further (e.g. sphinx). All results could be placed into another GIT repo to make them both revision controlled and serve as download site.

If we set up the production inside the GIT host as pipeline, we could even make it build with each push to the repo containing the PIM.

The only crucial point of this scenario is the transformation from the outputs of the modeling tools. The rest might be set up within a few days/weeks depending on the availability of people (probably me).

Flavio and Achim might have an opinion, if it is necessary to have the model split into several files for convenience inside the modeling tools.

Not so crucial for the workflow in general but more for the documentation part is the fact, that the classes would need to get documented very detailed. I am not sure whether the UIs of the modeling tools are sufficient for that. We currently have the full set of information inside the ownedComment nodes in XMI so that we could check this through importing and trying to edit.

But even this could be handled, if we rearrange the restructures text files for each class and package. If the UML just contains a brief description of the class, we would have enough documentation inside, to fill into the technical bindings and also into the pure “API” documentation. The detailed specification documentation in HTML and PDF that we create through Sphinx could then be separated into a set of files for each class. One of them is generated during every production run from the XMI and contains maybe the brief description, the field level information / documentation and links to / inclusions of the also generated graph, a restructured text file with the detailed description, another for examples… Those files would permanently exist within the GIT repo containing the product.

A simple check within the production line could create a table, for which classes any of the high level documentation files is missing.

Even bulk editing (for example renaming of field names in various classes) might be easier with this approach, as they can be handled with XSL which could be way more precise, as the actual appearances  would be addressed through XPath. I think, I could create a bunch of those transformations based on scenarios.

Best,

Oliver

Meeting minutes virtual MRT meeting 2019-02-06

Agenda as in invite of 2019-02-05

Goal: To have a plan for developing needed infrastructure and assign immediate work items per that plan

(1) Update on submissions to Advisory Group and NADDI Sprint prospects (Achim, Wendy)

(2) Infrastructure Topics

 - Modeling tool options & evaluation process (Flavio, Achim)

  Flavio and Achim have both suggested some tools, and there may be others. We need to identify a process for identifying options and selecting a tool.

  - Production Framework status & next steps (Oliver)

 There was some progress at the Berlin sprint toward finalizing a production framework. We need to get a status summary and then determine a process for identifying & making remaining decisions, if any. Note that there could potentially be divergences between the  TC  production process and the fast iterative process we use in MRT, if this makes sense, but that this may not be ideal. 

 - Testing Process/Recruiting Groups & Individuals (Larry, Arofan, Jay, Wendy)

  We have proposed a process which relies on having active testers, both internal and external. Some candidates are Larry's work with an R library and the ALPHA Network project (Jay and Arofan). We need to identify any other groups (at least a couple more are probably   needed) and determine how best to manage interactions with internal and external testers for supplying test versions of the standard and managing feedback (JIRA has been suggested - Wendy to comment?).

(3) Organizing groups for immediate tasks for Data Modelling and Modeling Issues 

(4) Plan for outstanding topics

    - Standards alignment/round tripping

    - Others?

Meeting minutes

Attendees:  Achim, Arofan, Dan G., Flavio, Hilde, Jay, Oliver

Apologies:  Jon, Wendy

(1) Update on submissions to Advisory Group and NADDI Sprint prospects (Achim).

The two documents ‘MRT - Modeling, Representation, and Testing Lifecycle: A Proposed Working Group for Building DDI 4 Core’ and the ‘NADDISprintPlanning’ were both discussed at an AG meeting on Monday January 20^th. In general there was a positive response to both documents by the meeting participants, and the MRT - Modeling, Representation, and Testing Lifecycle Working Group can now regard itself as an official working group of the DDI4 Moving Forward project.

The AG will send both documents to the EB. The NADDI Sprint Planning proposal will need approval for funding by the EB and will be discussed at the EB meeting on February 20^th.

Achim proposes that the new status of the MRT group is reflected with a new section on the Wiki, where the MRT meeting minutes and other relevant content can be published.

Status: Agreed to create an MRT working group wiki page with focus on the basic needs in the first version of the page and build as needs arise.

Hilde volunteered to put the basic page in place, and look into further options for the page.

In the context that the funding of the planned NADDI Sprint is pending, availability for a possible alternative one day meeting between a sub-set of the group members who are already at NADDI was explored.

Status: No clarification reached. The MRT group is waiting EB’s response on the NADDI Sprint proposal.

(2) Infrastructure Topics - Modeling tool options & evaluation process (Flavio, Achim)

Possible UML tools:

Flavio recommends  Sparx Enterprise Architect but is in doubt how close the XMI output is to Canonical XMI.

Achim explained about the UML tool Eclipse Papyrus and plug-ins. He also pointed out that we need to think of what would be the requirements for a similar tool. Achim means that Eclipse Papyrus has what is needed when it comes to management, creation and validation, but that Sparx EA has a better functionality. Sparx EA also has additional costs compared to Eclipse Papyrus, that can be an issue. A similar tool would need to be able to import Canonical XMI and a Canonical XMI should also be provided.

Regarding the output from the tool, both Sparx EA and Eclipse Papyrus both deal with deal with XMI export, but Papyrus does it in a cleaner way (EMF ecore), which is similar but not equal to Canonical XMI. Transformation to Canonical XMI would be needed for both tools. Eclipse has more plugins of standard based tools which allows model to model and model to text transformation. Flavio points out that this is one of the strengths of this tool. He mentions it is important to look into if/how the tool handles merging, hiding of attributes and a possibility to create views. Achim points out that a versioning technique is required (general comment).

Oliver brought in how a production framework could deal with processing of the model. It was agreed to wait to discuss the production framework until a plan to decide about the tool to use is decided.

Platform requirements:

The Canonical XMI was agreed as the single source of truth at the Berlin Sprint.

Achim pointed out the need to focus on requirements for a platform and procedures to maintain the canonical model and its expression as Canonical XMI. Two alternatives were mentioned:

1) Any UML tool could be used to play with the model (import to the tool in Canonixal XMI). New things are sent to a central modeler/team who performs a central editing of the model (requires that a tool is chosen a tool is chosen that can export Canonical XMI (requires some transformation work).

Larry points out that this alternative is too much risk on one person, while Achim means it has a risk, but also an advantage with more control over the model.

2) A tool with a central repository is used. Requirements need to be identified.

                Status: Agreement for alternative 1) for the immediate term. Find a person or small group that could be responsible for the central editing. Possible candidates mentioned were Flavio, Oliver, Larry. Explore alternative 2) for the longer term.

Follow-up tasks, Achim and Flavio:

• Achim will look into the XMI from Eclipse Papyrus and Sparx EA in comparison with Canonical XMI. He will get back regarding this in three weeks.
• Achim will also look into the model to model and model to text transformations possibilities offered by Eclipse tools, that allows to reach other transformations very fast.
• Flavio will look into Enterprise Architect and output.

Group follow-up:

• Find somebody (person or group) that could take care of the modelling edits coordination.
• Discuss production framework.
• In about three weeks, start decision about which tool would work best.
• Identify how to recruit further testers.
• Look into requirements for a centralized repository.

Appendix

Email from Arofan to the srg list 2019-02-06

Folks:

Just some thoughts following our last meeting regarding modeling tools.

I think it was Achim who pointed out that we need to understand our requirements here. As I see them, they are:

- Immediate Term: We need a platform for maintaining a canonical model and (at the point off hand-off to production) an expression of it as canonical XMI. This could be a single person in the short term, or a small group. As Larry commented, however, this runs the risk of becoming a choke-point in time, plus being a lot of work for one or a small number of folks, and potentially error-prone.

- Longer Term: It would be nice to have the model available centrally, so that all people could download it and use their tool of choice on it, to "play" with it. Canonical XMI sounds like it will act as an *import* format for this purpose, and Subversion (or something similar) would allow us to handle the management without too much pain. We would still need a gatekeeper or gatekeepers, but the demands on their time would be less, and the management of the model less error-prone. The requirement is for a format which can be used by everybody in whatever tools are selected, and which can be managed centrally, and ultimately expressed as canonical XMI for (at least) hand-off to the production framework (and distribution).

As Achim pointed out, the canonical XMI export support is a major hurdle, and makes the idea of agreeing on a single tool an alternative. The issues here are functionality and cost. If we use a single tool, then we can do collaborative modeling on shared files. The Eclipse model-to-model plug-in might help out as we approach this issue: if we can go between flavors of UML models, with canonical XMI as a pivot format, then we might be able to support the use of a number of UML tools more easily to meet individuals needs and tastes.

It seems to me that we need to do the following:

(1) Find a person who can act as a gatekeeper in the immediate term, and make sure that they have a tool which can produce canonical XMI (this may require writing a transform). This should meet our immediate-term goals.

(2) Formalize requirements for a more centralized solution, including (ultimately) a comparison of the single-tool and multiple-tool options.  Based on this, we could then answer questions about which tools are the best fit, including considerations about interfaces with the production framework. Ultimately, we need to identify and deploy the chosen solution, and appoint a person or persons as gatekeepers

It may be that we should ask everyone who has the time to identify what they see as requirements and envision a solution (or solutions) which they think would work best, to narrow the possibilities. Having these in written form would be great, so that we can easily incorporate them into some kind of draft document. 

There are a lot of theoretical possibilities here, but we are a small group with a narrow focus on the DDI Core, so we do not need theoretical perfection. What we need is a practical and sustainable solution for the course of the coming year, in expectation that we might need to extend it beyond that into the future.

Although this could be a very lengthy process, we should try to do this as quickly as reasonably possible. I see the decision on how to work (in the longer term) taking about a month to six weeks, ideally - otherwise, it is in danger of becoming an academic exercise with no outcome, as it has sometimes seemed to be in the past. If we can identify requirements and do whatever we need to do to deploy tools to meet them in that time frame, that would be really good.

Cheers,

Arofan