April 2024

Paris 10^th April 2024 10:00-15:30

COSMOS website

Highlight notes from EDD22 meeting

Agenda

Attendees

Introductions 

• Name and job title - see above.

• Organisation and country - see above.

• Experience with questions and questionnaires -  the group had a mix of backgrounds and DDI experience, from beginner to day-to-day users.

• Lifecycle or Codebook or CDI? – most of the group had experience or knowledge of DDI-odebook or DDI-Lifecycle, with a couple more attendees having experience with DDI-Lifecycle

• Tools - The two main tools being used in the group were Nesstar and Colectica in equal numbers. In addition, CLOSER and INSEE have developed in-house tools. 

• Ice breaker (15 min) 

Context and background of questions and questionnaires and the purpose of the meeting - see Hayley’s slides Questions-overview_hm.pptx

The presentation was followed by some clarifying questions about the differences between DDI-C and DDI-L. 

There are different ways to organise information (instruments, questions bank).

DDI-L might be daunting for organisations beginning with DDI, because there are more things you can do with it in comparison to DDI-C.

It is difficult to reconstitute a questionnaire from DDI-C.

The name of a question isn’t specified in DDI-C, and isn’t the same as the name of a variable, DDI-L can support this as it manages questions separately.

The relationship between question and variable is not always one-to-one, but can be one-to-many.

DDI-L supports the documentation of entire questionnaires and also supports images and stimuli.

Using DDI-C to document questions is better than not documenting, see World Bank as an example. 

DDI-C is less sophisticated than DDI-L, but that doesn’t mean it’s not a better solution in some circumstances as it is a simpler low-tech solution (e.g., for small and simple questionnaires). For larger questionnaires, the complexity of DDI-Lis important, but it is more difficult to implement and harder to convince others. We may need to think about how we communicate this. 

DDI-C isn’t ideal for longitudinal questionnaires because you only have information on questions embedded in the variables, but you can link questionnaires across waves in DDI-L which has extra relationships and references that you can build in (e.g. repeated questions). DDI-C doesn’t deal with this currently but the new version (DDI-C 2.6) supports persistent identifiers at the variable level, so although it wouldn’t be as neat as DDI-L, it is possible to create references. 

What problems do we have when working with questions and questionnaires or what would you like to do? Jamboard (45-60 min). 

Below are questions which arose from the Jamboard and discussions that the group had based on these questions.

How to represent question evolution in DDI? e.g. referencing input from previous data collection (i.e. feedforwards). Some questions from one wave depend on what people answered in a previous wave, so we need to know if the current question should be asked and how it should be asked – i.e. filters depend on variables from previous waves of the survey. This is possible – the way this is described in the UK is feedforward data which can be put into variables and can be referenced in the questionnaire, like any other variable. The source will be a variable or the response to a variable, so it is doable and it would be helpful to describe this process. INSEE is already implementing this. This is about the documentation of this rather than the implementation. Question evolution is well described in DDI-L. With DDI-C, it is possible to use “notes” to mention filters but it is less explicit than DDI-L.

How to deal with questions that repeat, but the meaning/response domain changes? Thiswell described in DDI via the variable cascade.

Is there a way to capture additional information about a variable when the information for the questions and its metadata are not available. Sometimes questions don’t capture all the information about a variable that is needed for someone to use data. Things like that concept could help to build a bigger picture. The group referred to the DDI Item type - Variables web page DDI-L supports a lot of information on the variable including the question but also other information (e.g. universe, concepts, analysis unit) which could add additional information to the variable.

There are no expert guidelines for building questionnaires. Guidance on what are the different choices and the trade-offs when designing and documenting questionnaires is needed. What you should do depends on the organisation situation, complexity level, and goals. A good practice document is needed. 

How to capture question/questionnaire metadata in DDI-C for it to be future proof for moving to DDI-L?How to best use DDI-Ci to allow migration to DDI-L in the future DO we need standardised guidance for this? It is possible that you can   document the information that is an captured as structured text inside thenote field, which would allow you to capture the information without the overhead of managing the questionnaires in the same way as you would in DDI-L. CLOSER initially thought they could transform metadata from DDI-C to DDI-L but found it was not straightforward. Do we need to add new items to DDI-C to hold additional key information, add structured notes, , or accept that there isn’t a good migration pathway from DDI-C to DDI-L. 

The cost of adopting DDI-L is steep (tools, learning the standard, capacity etc). It’s an investment to learn but worth it for futureproofing – how can we communicate that message? There needs to be a simple way to reach other people in the community, not only around tools but the standard itself and beyond the userlist. We need to communicate the advantages, something to show users how/where to start and have use cases or examples. There are some use cases on the DDI Alliance website, but these may not necessarily be kept up to date. 

DDI-L does not have the ability to support a way of expressing a "master question" - do we need a conceptual question - like a conceptual variable? . This was raised in the questions and questionnaires meeting at EDDI22.An example of this is when you have questions which are the same but have slightly different words because of the different data collection mode – there’s no way of representing that they’re both based on the same question (i.e., there is no abstract version of the question). This also relates to standardised scales (validated instruments), dynamic creation of questions,  or question banks where survey managers select questions.

Need to document in a multilingual way. This can be done in DDI-L, but how can this be done for questions in DDI-C?

Terms of reference 

• Relationship to other DDI Working Groups (WGs) and Technical Committee 

Which DDI Working Groups do we expect to work with and how? The different WGs were outlined and the process for how (WGs) interact with the Scientific Bboard (SB). – If the group produces any outputs they need to can be reviewed and approved by the SB. The DDI Executive Board's scope is related to high-level strategy and funds.

A lot of the comments in the earlier discussion highlighted a need to provide training and share knowledge. For training, would this group produce materials or ask the DDI Training WG to produce materials? One option could be the group would create the ideas and the content, and the Training WG could put the ideas and content together.  Since this group would be formed of people who are actively working on questions and questionnaires then it would make sense for the group to provide suggestions to the relevant WG, such as new training materials or  controlled vocabularies. 

The paradata WG could be related to this WG as they have a focus on documenting metadata around data collection (e.g. time to complete a questionnaire).

The group discussed the importance of communicating with other WGs t such as the Training WG to make sure there is no duplication of work, or when new features or updates are needed. There might need to be good liaison with the key WGs, whether that is an official person who is in both WGs, or whether this is more informal through members who happen to be in both. Planning the work and communication will be key. 

The group discussed the need to  produce its own outputs, but if there is already a WG that covers that, then the  group will liase with them, as to which WG is best placed to complete the work (this will depend on resources and expertise)  . 

As part of the group, there will be meetings that would be good to attend (e.g. annual DDI meetings), producing the annual report, reporting on what you’ve spent funding on etc. The group needs to produce a mandate and objectives and submit it to the DDI Scientific Board who take all the objectives and make a scientific working plan. The scientific body provides a bigger plan of what they want to achieve, so the group will have a better idea of how their objectives fit in and what they should be focusing on.

Working Group topic (30 min)

• Broad scope and purpose 

The group discussed the need to be a central point for responding to questions and putting people in touch with the right people, such as people who have experience with particular tools. The group should be visible to the community as experts in questions and questionnaires.

The group should determine  who is actively using questionnaires , what they are using it for (e.g. social science, statistical surveys, measurements), what tools they are using. and issues they might have with them in order to get an idea of the landscape. In addition, who would like to use questionnaires, and who was, but isn’t anymore. This would help identify possible technical issues, key challenges and barriers which can be discussed, and bring relevant people on board into the group. This could be the first task of the WG to determine the current landscape by circulating a survey to the community, this would also provide a list of experts that could be called upon.  

If the group identifies what tools people are using, this can be added to the DDI tools page, and it could be the group’s responsibility to keep an eye on and update the questionnaire-related tools.

If organisations aren’t using tools, the group could determine  the other ways  questionnaires are being created, for example through questionnaire creating software e.g. Redcap, and so there may also be conversion tools which we need to capture. This links to the DDI Developers WG. One idea could be to do a developers hackathon about questions to build links between the different elements of DDI around questions.

The group will cover all DDI products, The group could be a hub between the different products, concentrating on questions, measurements and questionnaires. 

In summary, the group would initially like to: get an idea of the questions and questionnaires landscape (including tools which are in use);  assemble a group of experts on the topic; understand the requirements of the community and then identify possible solutions e.g. training, documentation or updates to existing products, whilst working with the other relevant WGs. .

Terms of reference 

• Expected longevity of the group

The group is anticipated to be a long-running expert group, which could transform into something broader on data collection in the future, in addition to questionnaires.

• Determining chair

The group agreed that it would be good to have a chair and a co-chair to share the workload and responsibility, and how well this works in the Training WG. JJ suggested someone from INSEE would be well-suited given their experience and expertise in the field. Romain volunteered to be co-chair. HM agreed to be the initial co-chair in order to set up the WG, but will think about this  as she is still  currently the co-chair of the Training WG. The Training WG, SB and TC tend to change chairs every two years, so this would also be a practice this group adopts. Note that it’s helpful to have a year overlap between co-chairs, so there is always an experienced co-chair

• First meeting, frequency of meetings

The group agreed that meeting every two months for 1-1.5 hours would work well, and meetings could be scheduled in advance. The meetings will be online, but once per year the group could meet in person at the EDDI conference. Note the EDDI conference is also a good opportunity to disseminate and communicate outputs from the group. The first meeting will take place at the start of June, and ACTION: Romain will send a poll and organise the first meeting. In this meeting, the practical and operational aspects will be discussed in detail including setting up a regular time slot for the subsequent meetings  and where resources are kept. Note the usual setup is Google Drive and Atlassian wiki.

What does establishing best practice look like? / what are the current needs?  

Creating guidelines for beginners and experts was flagged earlier on in the meeting. 

• For beginners, it would be helpful to have information on the different DDI products and what you need to know about them to start building questions and questionnaires. 

• Information about design options and possibilities, and what are the advantages and disadvantages of the different options. 

• Information on how to think about the structure of a questionnaire. 

• Beyond this, guidance on how to manage and reuse the questions. 

• The guidance could include some kind of visual representations (e.g. UML) or use cases of what best practice would look like.Possible guidance on the minimum metadata requirements for a question so that it’s more interoperable.

• Guidelines or help with working with other standards. 

• The creation of DDI profiles as examples or case studies. There are already CESSDA developed/endorsed DDI Profiles for the CESSDA Data Catalogue and EQB: Profiles .

• There are not enough people comfortable with DDI Lifecycle which has delayed development of the CESSDA EQB – this is something the group could also engage in/provide support on.

• Wrap up and next steps

  • All - Minutes/ notes, which can be used as a starting point for the mission statement (directly in the Google document)

  • Romain - Poll for next meeting and organise first meeting

  • Becky and Lucie to add notes to help inform the group’s scope Hayley to finalise notes and share these with this and the wider group

  • Romain to email Hilde to propose this new WG

  • Romain - Create agenda for the next meeting