Earlier Meeting Minutes:
2022-2023 Minutes Page, 2020-2021 Minutes Page, 2018-2019 Minutes Page, 2016-2017 Minutes Page, Pre-2016 Minutes Page
Expand |
---|
|
ATTENDEES: Wendy, Dan, Jon, Darren, Flavio Implementation Language meeting around EDDI Funding for Fall meeting on Implementation Languages has been approved. We need to develop a more fully developed outline for this meeting to discuss with CDI later in July. Week after would be better due to room availability EDDI Right after Thanksgiving (Wednesday-Thursday) Hackathon (check with Ingo) More generic meeting on the Friday for broader contribution and gory details the next week (3 days?) Identify specific implementations What needs to be done in a consistent manner What can be flexible Explanation of usage needed
Codebook 2.6 review COGS No Jon next two weeks - no actions on COGS until then August meeting of TC in Minneapolis Funding has been approved Specific goals and outputs from August meeting: --Document changes in structure --XML, RDF, JSON, UML/XMI (canonical), Sphinx and restructured text documentation --Rules and output where possible - reconciling between the schema and outputs --when we cut the cord that we are able to get back to the 3.3 schema --What the different serializations look like - just doing vanilla transformations are a bit pointless --Technical review - can roll out serializations separately --XML and JSON are currently the most used and most stable and are probably ready first for review --RDF and UML/XMI has been tested by EA --If we wanted canonical XMI we have to change output or use the translation tool used by CDI --Build in canonical XMI as an output (currently has 2 - Normative 2.4.2, 2.5 with diagram and diagram exchange) new flavor would be canonical --Discussion of TC future leadership and roles --Change log of XML structure between 3.3 and 3.4 to see what changed and why (supporting multiple serializations) --Talking about changes - tree hierarchy is still the same but there are other things that are being updated to XML centric (CHOICE models) - substitution groups, removing a few things that are remnants of 3.0 and 3.1 (Identification and Reference properties) CANCEL Next two meetings - June 30, July 7 |
Expand |
---|
|
ATTENDEES: Wendy, Jon, Dan S., Flavio, Darren
AGENDA:
--Respond to SB comments on Review Process for Official Technical Documents
Document was revised in response to comments from the Scientific Board and Franck Cotton
Recommended that this document be integrated into the Process Document for standards publication. Hilde and Ingo will be notified of update
--ID and Reference for COGS Jira Legacy |
---|
server | System JIRA |
---|
serverId | b62a0bb6-d990-3186-ba9e-0c3828bdee04 |
---|
key | DDILIFE-3703 |
---|
|
Some things have been injected during serialization. Should these be flagged in some way rather than have them just in code
The model definition itself doesn't need any knowledge so perhaps in the settings area rather than just in the code. Clearly shouldn't be in the model
This would be useful for documentation at the serialization level.
See issue Number DDILIFE 3703--Publication process for CDI - new date is mid-July
--submission
--parts delivered
--difference on contents to determine if additional review is needed
--clear statement of what members are voting on as this is a new product for the DDI suite
Place on future agenda - Please think about this so we can move through the process expeditiously
--DDI Codebook comparison to other products - work done by SND - how to integrate this into DDI work on comparison https://zenodo.org/record/6505238#.YqyJwnZBz-g
Place on future agenda
Expand |
---|
|
ATTENDEES: Wendy, Oliver, Dan S Updated on CDI status, Codebook review status, COGS input corrections, and LOD work at UKDA Discussed August in-person meeting including increase in air fares and goals for the meeting Focus is on output of COGS Produce implementations that can be used for a broad technical review of new XML structures and new implementations - this is a technical review of the viability of these new structures and identifying issues that should be addressed. Discussion of TC direction over the next few years - new members, leadership changes, strategic directions
|
Expand |
---|
|
ATTENDEES: Wendy, Flavio, Darren In person meeting If funded by Exec Committee the TC In person meeting in Minneapolis at ISDRI has been approved for August 1-5, 2022. A note has been sent to TC members confirming the dates. Presentation at Members Meeting and Scientific Board The Members meeting presentation is only 3-5 minutes in length. Focus will be on funded activities and Codebook work. Will get a short statement from Darren so that this work presented accurately. Next year focus of coordinating multiple implementations of products and progressing on the production platform. COGS decisions regarding Identification and Representation properties, use of citation Reviewed issues and had a brief discussion of properties Wendy will create Jira issues (relate to gitHub issues) so that a full discussion is recorded and retrained within the design issues discussed in TC. (GitHub is specific to the COGS implementation and this should reflect both the issue and decision). Google documents will be used to focus discussion and linked from the Jira issue (these will be downloaded as documents and attached when issue is decided). Members will be asked to review and comment COGS input issues (xml to csv) will be addressed as follows: Modify transformation program where easy to do (new columns on csv with default values where information is not available in xml, etc.) Export resulting csv files, edit, and reload - this is a one time import and the issues are primarily due to inconsistencies in the xml Focus on import transformations should be on scripts that will be reused (example: import of canonical xmi from CDI to create a COGS copy) XML will be modified for addition of managed representations for those few not currently available and inclusion of other physical data product content using name modifications for elements with same name in different namespaces. The intent is to support the shift to serialization and use of inclusion by reference and to provide all the record layout options for review without massive remodeling. Handful of complex choice nesting locations will be mildly remodeled and documented in csv files.
|
Expand |
---|
|
ATTENDEES: Wendy, Jon, Darren, Oliver, Dan S. Annual Report No additional comments, will leave available through Sunday and then send in Codebook Field level documentation HTML version has been completed Wendy will review Change Log and update, set up review page and create the google page for the high level documentation draft Review should start by end of May. This should last to the middle or end of August Anticipate easy review because there has been a lot of consultation - this should mean that developers can begin work using the Review version we some certainty
COGS work Reference: Reference properties outside of the identification is the issue Reference is an entity and will act differently with serialization - RDF is a URI, XML / JSON model Review specifics of properties - Wendy will document Option is to create Complex types
Identifiers: don't think there is any difficulty by serialization. Some have their own identification (URI in RDF) Drop the difference between Versionable and Maintainable. There is only one way to identify an item (URN, AGENCY, ID, VERSION) Additional properties Identification is injected during creation of serializations Another area of confusion for additional properties is the use of Name/Type/Description and Citation (currently used singularly or together in some elements)
CHOICE issues are being sorted PhysicalDataProduct Add NCubeInstance to the basic RecordLayout now in the main schema Make the others derivatives of the BaseRecordLayoutType with unique element names and move the content to PhysicalDataProduct This will get everything in AND let us use one PhysicalDataProduct to import
Representations Recreate the few representation types that are not in the ManagedRepresentations. This gives them the same structure and we can determine later if we want to continue this division between representations that have references to other objects or what.
CV and RDF resolver: The sandbox will be set up end of next week https://rdf-vocabulary.ddialliance.org/cv e.g. https://rdftest-vocabulary.ddialliance.org/cv/ModeOfCollection/1.0.1 |
Expand |
---|
|
ATTENDEES: Wendy, Jon, Darren, Oliver CV work UKDS are finalising server setup next week. Oliver & Darren working together to get content in once it is setup. Target by the end of May. DNS needs to be setup at ICPSR.
DDItoCogs issues from Virtual Meeting Standing on the Agenda Report for the Scientific Board Meeting Date for F2F Technical Meeting Codebook Review Field Level Documentation – Jon to do Documentation will be description of changes Full High Level documentation to follow
|
Expand |
---|
|
ATTENDEES: Wendy, Jon, Darren, Dan S. Update on CDI --CDI is planning to get the package to the TC on June 1st. They realize we won't be able to deal with it until after IASSIST, but given the history of delays over the years, this lets them put a "win" in their column by getting it passed on. Virtual Meeting:Topics to discuss: Substitution Groups: Chose specific substitutions to use and how to implement XML substitution groups - is there stuff we want to merge or can we remodel that There is no model for the DCterms terms, these have been replaced by primitive types
Reference Design rules (managed references rather than modeled) Things that derived from Reference need to be added in other ways Handling additional reference information in terms of the overall model as it has to deal with multiple serialization targets References - is there some content that we need to retain that is not there
Rational to document: This will simplify the structure for everyone - in the future Draft of what was changed References were converted from a type to an auto-generated
Organization:Monday Tuesday Thursday Can we have some consistency in the UML-XMI expressed by the canonical XMI What are the realistic possibilities of having a COGS maintenance pipeline for CDI Prioritization of outputs from COGS
Actions:Wendy will draft content on changes in the DDI Agency Registry for the DDI Alliance page, have Dan S. review it, post, and add information to the Scientific Board URN document Jon will add information from today’s meeting on the Virtual Meeting page including a draft meeting schedule
|
Expand |
---|
|
ATTENDEES: Wendy, Flavio, Darren Funding requests Codebook 2.6 Finalize specification (header update) and make available to Darren (for CESSDA work) Complete draft of high level documentation by EOM Get out for public review as soon as possible in May Publicizing DDI Agency Registry work Draft content for web site in easy to understand words regarding capabilities and options
TC priorities for next year: Implementation languages - do not mean to hold up work of specific groups but want them to be aware of the work so we don't design ourselves into corners or have decisions driven by single products 2.6 completion and publication Requirements management Production framework Mapping within DDI Suite - translation work Explore options for Codebook in terms of design rules (what needs to be backward compatible) Syntax representations issues - maybe put new ones out is a beta-like mode for testing and identification of problem points - both Lifecycle and CDI plan RDF representations in the first half of fiscal year
|
Expand |
---|
|
ATTENDEES: Wendy, Oliver, Darren, Flavio DDI Agency Registry publication/announcement Identifying technical contacts in DDI Notes from Jared's email Next steps: At its next meeting, the Scientific Board will review and finalize the draft definitions of Scientific Representative and Technical Contact Jared will contact the member representatives requesting names for Scientific Representative and Technical Contacts. Both are optional. We still need to figure out the optimal way of sharing those names.
Preparing for May Virtual meeting - what we need to complete in April SEE: TC Virtual Work Meeting - 2-6 May 2022 (linked from TC page upper left box of current activities) Items below were noted today and are on the page UML/XMI review of what needs to be capture Updates to the ingest program of known easy to fix problems Outline the discussion issues around complex choices Identification/Reference details
DDI Alliance resolution system for CVs and RDF vocabularies https://ddialliance.org/Specification/DDI-CV/AggregationMethod_1.0.html Darren talked to Miles and he feels its very straight forward beginning work on infrastructure the week of the 11th of April. The software PUBBY they were going to use is quite old - wouldn't really need if we use the CESSDA generated HTML https://github.com/cygri/pubby/tree/master/src/main Do we need a user interface for this - for general publication of CVs This can be done with html, skos, and codeList What would the user see if they followed a link. They would get the RDF or JSON structure. But the SKOS could contain the link to the html HTML as a mime type could redirect to the visual page The transforms occur in a bitbucket pipeline using XSL transform. This would run with free of charge run time (say once a month) Technical Design Specification Added link from TC page CURRENT ACTIVITIES box and from the original Resolution for DDI-CVs and RDF vocabularies
|
...
ATTENDEES: Wendy, Jon, Dan S., Flavio, Oliver, Johan, Daren
Agenda Items:
Worked on Funding requests preparing them one for discussion with CDI (this was recorded as changes in the draft documents found in TC Draft shared google folder.
Did a final review of the recommendations for technical document review. This can be sent to Ingo and Hilde for discussion in SB
Members were asked to look at DDICODE-85 regarding a new Codebook Tree document and add comments as needed.
Discussion of the Recommendation for Review of Technical Documents brought up some broader issues noted below. Some of these are broader than the purview of TC so they will be raised in SB or EB as appropriate.
...
Discussion of types of documentation, technical interoperability, semantic comparability, use case focus
...
Organizing information can be difficult to meet the needs of different people
...
Need to be clearer about what can be found where
...
Technical interoperability
...
Semantic interoperability
...
Clearer publication of what is covered and perspective - on product page
...
How to describe the decision making approach in using different features (CVs, Variable Cascade, etc.)
...
What is the line between "official" publications of a product and what are supplementary materials (like the training materials)
...
Guidance documents by users of DDI
Expand |
---|
|
ATTENDEES: Wendy, Jon, Dan S., Flavio, Oliver Reviewed current work plan through December 2022 CV work - Oliver has been working on translation of output. Needs additional permissions at CESSDA. Identified focus for July-Dec 2022 Identified activities requiring funding
May virtual meeting on COGS work: Outstanding issues: Substitution Groups - Physical Complex choice - dealing with inheritance hierarchy, comprehensive modeling of current information Consistent modeling across model -
What can we do pre-meeting Documents recently discussed in TC draft google folder: WorkPlan for July Dec 2022 Review Process for Official Technical Documents Integrated production framework Requirements and Process Management TC Work Plan 2021-22
|
Expand |
---|
|
ATTENDEES: Wendy, Jon, Larry, Oliver, Dan S., Darren, Johan CV documentation in codebook CV group has requested that there be a link from within the schema to the specific CV This means how we change for mid-release CV updates DECISION: There will be a URI to the latest version that's what would go into the documentation We can update only doing a minor update. If documentation of new versions of Lifecycle are being generated we can easily integrated into the field level documentation Is the SKOS a near thing in terms of exporting the SKOS we need to process. Oliver needs to look at it. DOI's for Best Practices and High Level Documentation Would prefer DOIs to on the HTML version of the Best Practices, or High Level Documents Citing a standard you reference the specific home page URL https://libraryguides.vu.edu.au/apa-referencing/7StandardsPatents cite standard number and publisher Issue is do we want to maintain the versioning or take snapshots and deposit it ZENODO Either we manage the versioning locally or we have to do the snapshot and copy to ZENODO Step this back to the question of what is the need for a separate DOI for these official publications related to a product. We need a clearer sense of the use case for this as well as the broader need. Doing DOIs requires that we managing versioning more than an internal log of update changes within the document. If we can't cite a document without a DOI then we need a DOI. File and management persistent identifier versioning situation ACTION: Raise a TC ticket and write down what we understand to be the problem One point about persistence if we make changes in reorganization of our website we need to recognize the problem of breaking links. Nail down what needs to be done for the documentation of the changes to the DDI Agency Registry both on the site and on the DDI Alliance page, then the announcement. Mixture of a user guide and best practices document. Item for the Virtual TC - spend about an hour on the agenda. Coordinate with the SB URN working group CANCEL Meeting on 10th March |
Expand |
---|
|
ATTENDEES: Wendy, Jon, Larry, Oliver, Dan S., Darren, Johan, Carsten (2nd half for URN discussion) Codebook: Reviewed changes to agent types, conceptualTextTypes and concepts - spreadsheet should be implemented as listed Enter these as a set for review, then enter documentation changes to 68 items. Note that 5 items need examples. Mapping: Johan - Project Shoc (social sciences and humanities open cloud) to create mapping to other standards and policies for conversion. There is a month and a half to the end of the project. Present to work to TC next week. Contact Flavio who will not be able to attend. Recommendations part: The first choice - is that registering every URN that is created? Resolving DNS service are just for service resolution an end-point in a port (has been there for a long time) HTTP service end-points (3 built in web resolution, DDI which is the individual, and set with all related URNs) No differentiation between global and internal
Carsten: What services should be operated to support URN resolution What should be provided by the DDI Alliance What should be provided by the member agencies Diagram of URN resolution works (diagram in the SB WG paper) Need to clarify what is already available Once you know a URN belongs to DDI you can query DDI Agency Registry If there is an established resolver used by agency A service record is only providing an end-point (service and port) There is nothing saying what type of service is being provided You'd have to know the common name for the web/URN/verbatim service record and could use the DNS records to look up where those resolves are Looking of service look-ups is not well supported across the board like by web services How to come from DDI to the actual object "N to T" arch id, handles, etc. at California Digital Libraries This can do "N to T" Have to know what the No common resolution but there is a way of figuring out who owns the URN space and linking to the http://registry.ddialliance.org There is a common system https://registry.ddialiance.org You need to know the syntax of the DDI URN to proceed to go from ARPA to DDI (the registration in ARPA does that) Don't want implementers to use that API to resolve everything, but to use that as a means of structuring local tool so that A load limitation precludes having this available for handling all queries So what's missing? Identifier resolution and identifier end point. Individual identifiers can be resolved in terms of a consistent pattern within an agency or sub-agency. Individual variations from the pattern are not supported. Published content owned by the DDI Alliance (primarily CV's and RDF vocabularies) - Darren and Oliver are completing that General Discussion Access to the objects is in the court of the Agency In the recommendation (1) suggesting that the Alliance would register individual URNs (long paper)? If a DDI URN is found in the wild the DDI Alliance can direct you to where it is found That you can get a formal description of service Individual URNs were not considered an option There is a template approach (end-point at an agency level) they can provide additional breakdown at the initial endpoint (for example based on knowledge of internal structures) What the current service does not provide is validation of agency assigned IDs. It is their responsibility to state it is not a valid URN. Even with handle service you have to keep up-to-date with directions Use case not covered - is the Alliance taking on what is actually the agencies responsibility. The http://registry.ddialliance.org is simply a passthrough. Is it clear to the agency that this is what they'd have to do. What about a scenario where someone registers an agency and then deposits it in an archive. They'd keep their same agency ID and can define their own resolution. You could put the archive as the service for their agency. The resolution service would have to direct it to a differential agencies. Best Practice around this? If it is a distributed resolution environment and an agency has deposited in several different archives. They need a resolution system to address that. TC is open to further discussions with Carsten or with the URN Working Group as needed.
Background Information: Carsten's email: My understanding when taking on the lead of the URN TWG of the SB was that only the first step, Consumer<->DNS was working. I understood our mission to clarify which of the other exchanges in the diagram needed to be implemented by either the Alliance or an Agency. This is what our discussions on centralised vs decentralised solutions were mostly about. From Dan's slides from EDDI (https://doi.org/10.5281/zenodo.5747652) I understood as follows: Slide 8 says that the second step consumer<->registry for SRV records has been available for some time, but I don't know how it works and can't seem to find the documentation. Slide 11-1&2 allowed me to do the third step consumer<->registry/Agency for the URL of a concrete URN, but Slide 11-3 did only allow me to retrieve the pattern via API, not the specific answer for a given URN. The final step consumer<->repository is beyond the scope of the URN service. Did I understand this correctly and where can I find a full documentation of steps two and three to follow end to end? If I understand correctly, there is no mechanism foreseen to decide whether any URN is valid by the registry, but it would always return a URL that the repository then can't answer. (see screenshot) I am happy to discuss this also tomorrow afternoon and please do forward this mail to those whom it concerns. |
Expand |
---|
|
ATTENDEES: Wendy, Larry, Dan S., Oliver, Flavio, Darren XKOS best practices: XKOS will have an invited technical review starting Feb 25, followed by a public review for comprehension lasting 6-8 weeks Publication will have a DOI (working with Jared on process) Wendy will draft a Guidelines for officially published documents (Best Practices, high level documentation) for submission to the Scientific Board for comment. This will be based on the XKOS process as the first document published under this process.
Codebook: DDICODE-83 and general policy for use of conceptTextType Result of this is that ProcStat will not change as the only real use is internal processing information and those with processing systems have not requested this. There are no broader standards. Other items in document were approved.
The cutoff for deciding if an element should change from a simpleTextType to a conceptualtextType: DDI Agency Registry: Link to presentation now available. Contact Barry about marketing announcement. Dan S. will work on working with Barry. Wendy will get these new capabilities into the URN Working Group paper to make sure they are complete Wendy will review directions in Administration section of DDI Agency Registry and write up an information page for the the Agency Registry page on the DDI site. Content will be sent to Dan for review when ready.
|
Expand |
---|
|
ATTENDEES: Wendy, Jon, Oliver, Flavio, Larry, Darren, Dan S. Update on Codebook activities: Review of field level documentation has identified one or two additional needed changes (for consistency). These will be grouped and entered for review as a set. Updated work on high level documentation - addition of a "tree" representation of Codebook
Follow-up on TC review of URN Working Group paper: Timing of review by TC will be discussed in next URN WG meeting Wendy will update the URN paper with examples from Colectica Presentation DDI Registry presentation by Dan 10.5281/zenodo.5747653 https://doi.org/10.5281/zenodo.5747652
CV LOD system SKOS recommendation There are also documents on the Atlassian site. These will be merged and TC members informed that it is ready for review and comments Versioning approach is being worked out and entered Transform of existing SKOS into a cleaner SKOS (Oliver) Publish transforms into bitbucket and then those will be grabbed and posted to the cloud 2nd week in march is slated for setup of the cloud platform New CV tools also have been proposed to be deployed in March Oliver has been in contact with CESSDA so that he is aware of where to do tweeks in URIs to deal with identifiers for codes
COGS After the virtual meeting regarding COGS input we will be focusing on COGS output with a goal of identifying where similarity in formats for XML, RDF, JSON-LD, other formats is useful between products and where differences are needed to support specific applications. Planning needs to start for funding and scientific plan in conjunction with the groups developing products. For example, Pierre Antoin has a relationship with Scientific Board in terms of development of RDF JSON-LD for DDI-CDI and RDF in general. This discussion is broader than TC and work has been done in each of the products with one or more formats that needs to be considered: DDI-L - XML (moving to full serialization), plans for RDF, JSON-LD, UML, etc. DDI-C - XML (hierarchical structure) XKOS - RDF (SKOS extension) SDTL - JSON Controlled Vocabularies - RDF (SKOS), XML (Codelist) DDI-CDI - XML (serialized), plans for RDF
Google groups and role of SRG list Update on Google Groups and continuation of the role of the SRG list to serve as a means of keeping interested people in touch with development discussions. Dan had question on specific feature of set up regarding prefixes. He will contact Jared directly
|
Expand |
---|
|
ATTENDEES: Wendy, Oliver, Larry, Flavio, Dan S., Barry Codebook - LOD Infrastructure Met with staff and what do with the URI's released by the CESSDA service CESSDA is doing a relaunch in March with minor changes which would cover all the system changes we needed for URI scheme Bitbucket repository will be set up soon so Oliver can pass over the scripts for production We may not need an extra transformation for target SKOS 2 major fixes are URI's to be really resolvable and not deliver empty descriptions for catagories if they don't occur in different languages URIs are generated with CESSDA output and we would be able to define the URIs produced Categories would have a consistent technical identifier rather than a clear text identifier SKOS to Codelist - Oliver will see if Darren has done any work on this, if not he will do this within the next week or so
Virtual Meeting April meeting - Dan and Jeremy busy the week of the 4th (6-11 unavailable) XMI/CDI: Cardinality of both source and target Relationship types - aggregations, compositions may not be relevant Uses and dependencies (patterns) Types are in the names now Achim has done drafts of description of UML that can be used for this style of modeling We would want options for getting this into COGS Review of CDI XMI and of Achim's draft - Flavio and Larry
Nail down dates as soon as possible so we can get some time frames for pre-meeting work The mix of data and metadata is important - JSON is a good vehicle
XKOS Best Practice Review Scientific Board WG on URN - Beneficial to make sure it addresses what is currently available and what is still needed It can resolve you to an individual URN - they can define a range of services Content hosting is not supported - URN resolution is solved TC should provide feedback on the draft of the document as this is the group that has been working on it for the past decade There are a lot of resolution options already available and we'd like to make sure they are focusing on new
|
Expand |
---|
|
ATTENDEES: Wendy, Dan S., Larry, Johan, Darren, Flavio CV work Discussion of low level specifics took place this week. Oliver expects transformations complete in a few weeks and Darren should complete some of his work. Something testable by end of February. CESSDA SKOS compliance probably not until end of year. We are not dependent upon CESSDA compliance to complete our work. CODEBOOK Get a 2.6 out for review Work on weighting is not ready yet and should not hold up 2.6 given the need for DataVerse related. Better to get it correct than to put it in not quite finished. Backward compatibility rule. Don't hold but encourage movement on the sections that have been dropped (SDI and NCube). DDICODE-39 is another one to hold for further work. Note these issues in the release for review information for 2.6 and add a label post2.6 to these issues. Larry is looking into DDICODE-39 to see if there is a clear and unambiguous solution that could go into 2.6. If not it will be worked on for 2.7 Set up high level document site for Codebook with Jon.
CDI is still finalizing documentation and will get to use dependent upon peoples work availability. ADDITIONAL ISSUES we need to organize for discussion: XML is currently expressed in different styles between Codebook, Lifecycle, and CDI RDF usage is currently in SDTL and XKOS Implementation languages for representations
Discuss when work plans need to be in for SB and how this coordinates with the financial requests. NEXT WEEK NO TC due to Scientific Community Meeting |
Expand |
---|
|
ATTENDEES: Wendy, Larry, Oliver, Dan S., Darren, Flavio, Johan Requirements and Process Management - Integrated production framework and platform https://docs.google.com/document/d/1TEy2zdxfARDgkOABb6kzuAmHK2oBJ0f5CmxL9F9pEbQ/edit#heading=h.avtxoxjt5vfy Good idea to have a sense of an integrated platform Infrastructure documentation - designing a standard SDLT We need to have a person who will oversee the design and implementation of the infrastructure Inventory of the existing buckets There shouldn't be any piece of production process or work we do that isn't in a public space Shouldn't rely on code or procedures that are in someone's head rather than in some published location We don't want systems reliant on an individual Availability of platform, both software and hardware, (needing to run on individual computers and requiring set up - example documentation set up for field level documentation) Initial discovery exercise - spreadsheet Wendy set up Look additionally at options If we need additional cloud space we will need to request beginning in the new fiscal year FFair(?) is set up to rebuild documentation each time a change is made Identify what is used by each product When you use free tools is everything exportable? yes, it just runs a rebuild script in an conformable environment with multiple platforms https://ci.appveyor.com/project/ddibot/ddimodel Darren will take a first stab CDI Jan 19 would be earliest but they are still working on the documentation so will probably be a bit later
Codebook Will be scheduled for next week to see finalization for review CV work Progress has been made and Oliver and Darren will meet next week Versioning at CESSDA is getting clearer (meeting on Monday) Profile of SKOS structure
https://ddi-alliance.atlassian.net/wiki/spaces/DDI4/pages/2766569473/Technical+Design+Specification#Metadata-Profile-(CVS%3C%3ERDF%3C%3EDDI-L) Requirements and Implementation Management Needs additional work - need to determine how to go about this DDI Agency Registry update work Acknowledged that Colectica has completed the work on the DDI Agency Registry update and presented on the work at EDDI. Announcement will be made when video of the presentation is published so that it can be referenced in the announcement. |
Expand |
---|
|
ATTENDEES: Wendy, Jon, Dan S., Oliver, Larry, Barry Review of Work Plan: XKOS: Talk to Franck and settle on a process Light way would be to do it to the list - set up a comments process Comments period for Best Practices Meet XKOS needs and outline a basic approach to formal Best Practice papers
COGS Virtual meeting Jon will drive meeting Correct the input issues we know about Focus on input accuracy, support for output needs in XMI, input of XMI as well as current XML content, sorting through entry issues and complex nesting Easter is a problem week before or after 17th April (Oliver is available) Oliver may have some problems with the week of April 25 Google docs for document working BitBucket features - due dates comments Slack possible but may not work for all Task and process assignments Who we want and what we want to get done, set up agenda early so we can nail it down early
Mapping Work: Additional Notes: EDDI videos will be coming out in the next few weeks as they come in - Jon will talk to Jared and probably post by track Follow up with Jared and Exec on the period of the fiscal year given shift in work plan year to January/December (this year’s work plan goes through December 2022, Dan, Oliver, Christophe, Jared, Jeremy AWS and Drupal site interaction and support Recap: some sections will move to AWS (specifications) remaining will be moved to wordpress Is it wise to split this down the middle and move to 2 sites. This might be difficult to split correctly and to maintain in the future.
--Is it possible to move everything to AWS? How complex might this be to maintain and supervise --Need to move away from Drupel as it is not supported by ICPSR and want to move it to a vendor --Fee for moving it over and an annual maintenance amount ($3-4,000 a year) --Urgency because support for their version of Drupal had gone away, did get year's reprieve --In the mean time the AWS option arose and discussion of how --Lots of static content that would just be moved --Some functinality - bibliography that is maintained in an ongoing basis; posting news etc. --Happy if it did not got to vendor --Can we support on-going posting and maintenance --If not, we can still go to WordPress keeping basic content there and other on AWS Current view of what goes to AWS: (1) Content under the following virtual URLs will be moved to the new UMich AWS instance ď‚· https://ddialliance.org/Specification/* - static content ď‚· https://ddialliance.org/products/* - static content ď‚· https://ddialliance.org/learn/resources - single static web page ď‚· https://ddialliance.org/learn/resources/ddi-profiles - single static web page ď‚· https://ddialliance.org/resources/tools - single static web page with filter o Also move relevant referenced items under https://ddialliance.org/tool/* ď‚· https://ddialliance.org/learn/markup-examples-by-version?field_ddi_product_tid=All o Also move relevant referenced items under https://ddialliance.org/sites/default/files/* ď‚· At the root level under http://ddialliance.org . o https://ddialliance.org/Tables.dtd - downloadable document --Clean up even before vendor --Hidden stuff would not move and Darren's list may include that --Splitting may have unintended consiquenses such as loss of database functions (publications, examples, etc.) --Do we need all the functionality that is there (examples etc.) - if we add in with the vendor --Would the options we are looking support these --Issue about having to shadow the brand and sync it between 2 sites and with a 3rd party vendor --What potential costs there would be with backup etc. - there is still need for web support (not completely volunteer) --It seems to me that we are missing a written target sitemap (whether we have one or two platforms) that would guide the specifics of the migration. --Could we take a few weeks out to design what we want the site to look like Include Training Group We have a current website that needs a spring clean but we have no idea what it should look like Can we take 4 weeks to say this is where we want to end up (google doc) - we don't have much written down on this at the moment Drupal to WordPress is somewhat automated there is also a lot of sorting and hand work to make the move. https://docs.google.com/spreadsheets/d/1y54OKTmJeSV3YwxR6RBa7yJyHvD8P1drjPmG4C2vZoM/edit#gid=517461823 Lists out the sections and owner groups Detail gets complex in the products sections Resources has some sections What exactly do we have - can we flatten it out? ACTION: We should look at what we want rather than just what we have Create small group to look at this: Jared(co), Jon, Darren (co), Wendy CDI technical review kickoff Flavio looked at it No draft of production process review issues - get from Arofan (send him a draft of request and list of people we've identified) Automation is still lacking so how is that evaluated We did write to them some time ago regarding - dig out letter we sent to them and verify all the parts are there Asking more information on production process itself; make sure its available Jon will look next Friday Funding requests see google sheet: https://docs.google.com/document/d/111V1om6vbaJQCgxYI_znA03xDWW9HwW0iMGHxNVVnHU/edit?usp=sharing Workplan for coming year Review workplan for tweaks Updates on feedback for DDI LIFECYCLE V.4.0 BETA No only had one piece of feedback from Romain BETA 2 would be good to do when we have the ordered flags
|
Expand |
---|
title | 2024-04-11 NO MEETING |
---|
|
|
Expand |
---|
|
ATTENDEES: Wendy, Jon, Darren, Flavio, Christophe, Dan Pages from Alliance site We need to clarify what is moving from the current alliance site before it shifts to WordPress The Alliance wants to keep this site a simple as possible and there are currently some pages under LEARN/Resources that have underlying databases The majority of content we are looking at is static and is under Products Everything that is under Specification is easy but need to determine what needs to be moved The real question is about being able to identify specific nodes that can be handled by the remote site (RDF resolution system and CV services on UKDA site plus new DDI Alliance AWS space) One possible way is just a lift and shift move all the /Specification Landing page on WordPress site and then it sends you off to other system What Darren needs is a site map (node list) so he knows what gets moved
Specific questions: docs.ddialliance.org - what is this intended to provide Currently we have a number of official documents associated with specifications that provide access to documentation such as field level documentation and high level documentation These are static sets and can be moved If it is difficult to tease out the nodes it may be possible to support static content under Products - all of this content is managed by TC
Resources: Tools, examples, profiles currently have underlying databases and we would like to retain this functionality These items are more in line with the products and it may be optional to move them under the Products umbrella. Learn could still retain a page that would describe and link to these resources To accomplish this we need to locate a workable opensource tool to manage search and display of these databases. TC will begin this search
Agency registry: ACTION ITEMS: Obtain site map (node list) Transfer page listing for move to Jared’s spreadsheet Determine content that must move to retain or add functionality or to just support easy division of contents for routing Clarify who will physically be doing the set up on the AWS configuration Identify an open source tool to support access to database content for tools, examples, and profiles Determine consequences of moving content to alternate site based on node divisions proposed
|
Expand |
---|
|
ATTENDEES: Wendy, Jon, Oliver, Darren, Dan APOLOGIES: Flavio 1-Codebook pull request ok and merged 2-Funding requests (Deadline is 26 of April) Face-to-Face again Infrastructure costs - flag and could suggest it should be a regular budget line; software contingency budget as well as on-going costs 10K last year was infrastructure contingency from which a couple of months of AWS Alliance pays Colectica and this will move to AWS at some point As part of the move to AWS a bit of work needs to be done to convert from zone file to AWS (route 53) - Dan will write up Admin data meeting - talk to Flavio about meeting next year; funding request
3-Strategic Plan Looks good - no questions or specific concerns from TC members at meeting 4-Administrative documents Not ready for discussion 5-CDI submission to TC and production process review Darren talked with Arofan about RDF URLs and everything is in line Other: CV work is done ready to make the change after confirming a few minor points with ICPSR and the CV group https://ddicv.ukdataarchive.co.uk/ new format for the CV page (domain to be changed obviously) No ISI proposal but will look at the IAOS conferences coming up
FUTURE: No Oliver until April 18 No meeting on 11th due to COSMOS conflict |
Expand |
---|
|
ATTENDEES: Wendy, Dan, Oliver, Darren, Christophe AGENDA Resolution system CV system RDF hosting/resolution RDF hosting/resolution for CDI (upcoming) is not a dependent on the CESSDA CV version issue being fixed. DB will contact Arofan to clarify CDI RDF serialization’s readiness and double-check required URI patterns for CDI All CVs have unique URIs based on version numbers and are stored in Fuseki default graph. For non-CV products e.g. CDI or DDI4 (that don’t necessarily have versioned URIs) we will create individual graphs for each product.
Codebook pull request review OK and merged 4.0 BETA issues - Comments also added to issue discussion in GitHub Item 8 Changing from internal CV to external could be a Lifecycle 4.1 task Identify those that could be CV Add enumeration documentation to the item type Group ones should probably be taken over by CV group Dan will look through to make sure enumerations are coming through correctly - OWL specs may require URIs for enumerations (this would be an issue for next round of review - how referenced Wendy will look at CV options now. Darren can look at later. This is not urgent. Adding documentation on lists that are currently there. Create notes of what should be added to each README file.
Item 23 Not critical in short term. Good to get multiple eyes on this. Item 36 Should be addressed in 4.0 BETA 2 Add additional notes on previous discussions Group instances to see what could be managed in the same way
FUTURE Administrative documents (Jon) as available (Jon not on call) |
Expand |
---|
|
ATTENDEES: Wendy, Dan, Flavio POST-MEETING WRAPUP (due to forgetting US change to DST): Jon, Darren, Oliver, Christophe Review of approach to issues 8, 23, and 32: Additional Topics: ISI presentation: ISI do we want to propose something on DDI like last year in Ottawa? Does TC want to send something - IPS We have 3 weeks - kick around an idea for a topic Something beyond use cases Parts of DDI - multiple serializations How content fits together and can move between products 3-4 people with or without discussant XKOS: Best Practice ready to go so look for email on that |
Expand |
---|
|
ATTENDEES: Wendy, Jon, Oliver, Dan, Jeremy, Darren, Flavio, Christophe Codebook Pull request has been amended and content is now correct Looking forward, the concept/conceptualText will be posted immediately following the resolution of the current pull request This will be followed by the remaining issues which are more targeted
ACTION ITEM: Dan will note items that are NOT documentation change so they can be reviewed next week DDI-L 4.0 Beta 1 Issues Another beta the middle or end of April after gathering COSMOS feedback An issue on representing sentinel variable values in Represented Variable All new content issues should go into GitHub. They will be tagged for releases. This will also allow broader discussion of content issues in clearer way than we have been able to do before.
ACTION ITEMS: Wendy: #36 - go back and pull together the earlier discussion of dealing with extended reference and review current need for continued use of this content #8 - review enumerations as converted to CSV and conversion to outputs #23 - locations of ordered attributes Jon: Administrative documents - add content on #16, #17, #30, #31, #32, #33, #34 |
Expand |
---|
|
ATTENDEES: Wendy, Jon, Oliver, Darren, Christophe APPOLOGIES: Dan, Flavio Webinar review A lot of people who we seldom or never seen Good suggestion: Missing documents What is the model Master? Development version
In terms of code the past published specifications can be compared Between 3.2 and 3.3 we could do a dif but we can't track between 3.3 and 4.0 because each generation is structurally different (dependent on mapping and change documentation) The ability to track individual changes How do we want to do this? If we publish 4.0 as a package At the moment published versions are in Bitbucket so we need a separate repository to hold published versions in GitHub to replace this Jon has started adding files (citation, license, contribution guidance, form for contributions One issue about having a missing license file we use cc-by [Note that the Alliance has determined that this is the license to be used by all products. We will need to check license on production programs (cc-by was not on the list of options] Document how things move through approval and that people can build on their own branch There is a GitHub project called DDI-CDI and DDI-CDI-resources Needs to be on DDI Alliance set of repositories There is no governance on the Alliance site
ACTION: Jon will start drafting governance documents Status on CV and pipeline Codebook Pull request for ddi-c_2 https://github.com/ddialliance/ddi-c_2 Note that the first ff3481b is bogus. It is the second commit 0cce353 that contains the documentation changes to codebook.xsd How to process through these? People could put in comments in before next meeting for the documentation The documentation content and the next (Concept/ConceptualText) are large. Some review should take place outside of meeting and then deal with questionable areas with group. Future edits should be more localized. Oliver will check to see if there is a clear way to correct and entry error (alt is to reject pull request, kill this Fork and start over)
NEXT WEEK: Codebook - Documentation review FUTURE AGENDA ITEMS (as ready): GitHub administration and guidance Publication repository on Bitbucket - options/plan |
Expand |
---|
|
ATTENDEES: Wendy, Jon, Darren, Flavio, Christophe, Jared (guest) APOLOGIES: Dan GitHub: ACTIONS: Need to set up an account on AWS so that we can begin organizing this Jared needs to set up the account with appropriate permissions Darren will ask Michael for the DNS zones Wendy and Jon will create a list of items that might move from ICPSR to this new site
Clarification of document repository approach ACTIONS: DDI-L 4.0 BETA review of webinar Jon has done slides In middle of the webinar Jon would like to demo a change on a fork and generate 35-40 people are attending This is a BETA and there are already errors which need correction Public review in summer at the earliest
Questions regarding spreadsheet ACTIONS: Add to key meaning of a,e,b in 3.3 column Finish sumamry of purpose of moving to Version 4.0 / use of COGS Get these posted and link information to Jon so he can add to slides (by Friday) What is the current status of CDI - finalizing documentation review checking images, Arofan is wrapping up the documentation and pulling together
XKOS update: XKOS best practices document almost finished integration of issues - probably out before COSMOS, maybe next week They will create an announcement and we can get this out and posted on the web page
|
Expand |
---|
|
ATTENDEES: Wendy, Jon, Darren, Dan, Oliver, Jeremy APOLOGIES: Flavio Infrastructure: Set up account - and we need someone to work with Jared In terms of setting up environment - need someone - Darren is willing to do set up Getting permissions to the right people Contact Olof and look for a person to follow-up and manage over time Apache server serving a static site Separate containers for different things - keep it simple cover docs.ddialliance.org instances AWS issue: Darren - we can get Simon 2 weeks in March 11-18th) to get the CV set up at UKDA Moving pipeline off Bitbucket to GitHub: Oliver has already moved it to GitHub and Oliver will send URL of repository https://github.com/ddialliance/ddi-cv The creation of repo content is still manual but should be turned into a pipeline in the nexts Sanda can check out the html specifications. Pascal can also do checks Webinar Spreadsheet 3.3 to 4.0 - post for TC and webinar support Jon has updated the presentation outline based on TC comments Add Demo of COGS showing how easy it is to make changes and then generate their content for testing Members planning to attend need to register FUTURE AGENDA: Next week: review of webinar in prep Two weeks: Check in on CV, RDF, and repo pipeline - sanity check |
Expand |
---|
|
ATTENDEES: Wendy, Dan, Jeremy, Flavio, Oliver APLOGIES: Jon DDI-L 4.0 BETA webinar As of last Friday, we had 35 sign-ups. Draft of short presentation contents https://docs.google.com/document/d/1jeqDxr_4P3HM-oeW4j-g-WYr3aAKQeUwNooyilgGWbY/edit?usp=sharing See document for comments AWS set up TC budget for AWS has been approved, so we need to get moving on setting up an account, presumably we need to get Jared to put the Alliance down to pay that. Uses funding requested this year for infrastructure support. Next fiscal year this should be proposed as a new permanent budget line for association. Talk to Darren and/or Olof about taking this on to get set up. Regarding cloud deployment. Whoever sets it up we should management deployment using build scripts. The CV stuff is set up this way and there is something to start with. We will need to add in things on the documentation. See how long Jared would need to set up the account. Be clear what needs to get done prior to setting up the cloud space and moving things into it. Product Page Implications for how the content is made available on the download and doc site for each product Currently use docflex for the Field level documentation of the XML. Should this be continued with the COGS product. XML specific documentation as well as language specific documentation need for field level documentation. Can we add to the doc site (links to specific output documentation) A revised example will be made based on comments. Follow with examples of transfer of all product pages. |
Expand |
---|
|
ATTENDEES: Wendy, Jon, Dan, Oliver, Christophe APOLOGIES: Flavio DDI Lifecycle 4.0 BETA webinar Went out on DDI Users Group and some socials Put a reminder out 2 weeks before event Check with Jared on numbers Jon will draft up something for next week
DDI infrastructure proposal Has contacted relevant working groups CV XKOS and SDTL are fine CDI group ok; some longer term issues that can be addressed as they come up in terms of production line Add some statement about governance around GitHub Can start setting up landing page for development work so we can inform people
Atlassian Some worries Many people are moving off it and this could cause the loss of open source support (due to lack of paying customers) We have moved off in terms of Bitbucket But we still have JIRA and Confluence Clean up JIRA in the next year to drop unused and archive old issue trackers Consider free support services we use to develop contingency plan across the board
NEXT WEEK: |
Expand |
---|
|
ATTENDEES: Wendy, Jon, Dan, Oliver, Flavio Discuss page change drafts for products (TC-242, TC-243, TC-244)
Product Page list of versions OK Get this done first as this resolves the Google reference issue The page of links for filing issues and how-to page are a good idea Product version page revision is a good first step by pushing critical information to the top and making it visible on the initial screen. Continue to revise based on information from developers/implementers and prepare updated draft along with an example from Lifecycle. These need to get updated prior to publication of CDI, new Codebook 2.6, and DDI Lifecycle 4.0. Issue to address in future: Current download packages are now served by an on-site (ICPSR) copy of the product and XML is provided for validation purposes through the ICPSR site. How should this change with multiple serializations (XML, RDF, JSON, UML-XMI, etc.)? The Bitbucket repository of published products currently serves as an archive and is not linked to the product pages consistently.
Organization and ongoing management of GitHub content This was just a heads-up on the type of information TC needs to organize regarding GitHub space. It is primarily management information (Guidelines for development groups and users, tooling pipelines, space management, constraints regarding triggers for moving from free to fee based services, etc.). This is still in a period where we are obtaining comment. These notes from the CDI meeting will be more formally organized and provided in writing by CDI. The Scientific Board is aware of this work and is primarily monitoring the process to ensure input from development groups affected by the shift to GitHub. TC pages under Learn dropdown on Alliance site Resources: We need to look at how examples, tools, and profiles are managed and organized in the new system. We will no longer have the on-site option of underlying databases. Primary decision is which, if any, of these should be managed off-site. Getting Started: See TC-245 for collecting information on determining the purpose of this page, what information should be provided, and how to present it. Members were asked to think about these issues and add comments to the issue so that we can move on revising this page in cooperation with the Training Group and probably future Marketing Group. This should be a priority. Jon will talk with Training leaders to get their initial thoughts. NEXT WEEK: Webinar for 4.0 Beta |
Expand |
---|
|
No meeting was held. The following is the information sent out to the group for their attention and action: TC issue tracker I have reviewed and updated all entries In particular see TC-242, TC-243, and TC-244 and add comments regarding layouts provided in attachments Use filter TC-outstanding to filter out completed issues Notes from DDI-CDI meeting on 2024-01-17 Comments about Git repository raised during CDI meeting on 2024-01-17 - note that Achim was not present so there will be more Git repository management - how does TC support product groups in this? Guidelines Informational materials Help identify persons if group is unable to provide this Deirdre is willing to be involved as setting this up Back-up people Git repository Tools/workflows Organizing BETA access, support repository, policy, pull requests Coordinate how to organize Working with input from outside of the group (developers work) Making clear what things are Granularity of pull requests (issue based) Branching - readme on how to do this, what is supported Pricing models for AWS (space is not the issue, but deploying own servers/code base) Each product needs to provide a canonical expression of the product CDI is the UML model expressed as canonical XMI Each product needs to be clear on this - how to visualize the model in the case of COGS base Canonical model has to be publicly distributed |
Expand |
---|
|
ATTENDEES: Wendy, Jon, Dan, Jeremy, Flavio, Oliver Update on CDI work Variable Cascade issues - providing guidance through webinars, documentation (it appeared there weren't direct predicates linking along the chain, appeared there were just a single predicate used for different relationships) Will this require additional review?
Topics for future webinars DDI-L 4.0 BETA review Supplemental content - changes in the content model; why and what was done; update DDI Lifecycle Best Practices (how to use some of the schema structures to address 4.0 in addition to 3.x best practices) Updating model - just raising issues and modifying at the end
EDDI session on priorities Follow-up on - web pages and webinar with Jared |
Expand |
---|
|
ATTENDEES: Wendy, Jon, Dan, Jeremy, Flavio DDI-Lifecycle version 4.0 beta follow-upWebinar of some description Q&A - Highlight of why we moved to this and how they can interract with it or Anticipation - presentation Q&A seems best approach A bit of introduction to version 4.0 (about 5 minutes) Back end of February would be good timing Discussion pieces out from mid-Jan to mid-Feb: get a list together and see what is important to front-end 3:00 UK (10 eastern, 9 central) - 21st or 28th of Feb (Jon will follow-up on this) It's about a small group zoom with google form for registration We've only really asked about OWL serializations Do we want something more on the model conceptualization? On the serialization things? Model is the same as 3.3 with only changes made to so only specific changes for choice and physical description
ACTIVITY: (Wendy review and create list) pull discussion pieces together, clean up, post, and send links Roll out discussion pieces. Product pages on Alliance Site Overview of products page can be updated but is in good shape overall https://ddialliance.org/products/overview-of-current-products Product pages: Less wordy More prominence for the access to the html documentation Update links Setting up pages that were held off the site (determine what should be off site support - RDF resolution All of the products have autogenerated documentation - move to a docs.documentation site and link from the main pages Examples, tools, etc are candidates for moving off-site Get comments from Jared (contact him for what feedback he's gotten)
|