What are we doing?
Data types or data sets can form the columns and we can complete information against the dataset.
Important to get to the questions that the researcher will be asking.
Cahllenges faced: accessibility - access to data and the appropriate tools to open different types of files. How do we match these specific data with the standards or specifications that already exist.
Fernando's spreadsheet.
MO. In some instances the data is downloaded and used. In others the data is dynamic and so refer to the website.
Units of measure.
Decision to go with the Ebola Case Study. What is the research question?
| Name (Working term) | Area of coverage | Demographic Data (Gridded Population of the World (GPW), v4 | IDDO CDISC Ebola Data | Humanitarian Data Exchange - Healthcare worker mortality |
|---|---|---|---|---|
Spatial and geographic | Information required for describing geographic information and services (e.g. ISO 19115) | Available in an xml format (FGDC) that provides bounding boxes. Granularity is an issue. | Clinical data collected from different organisations; across a number of hospitals. This is an assumption. CDISC data dictionary seems to only deal with country. Location of the clinic is important information, which is also potentially disclosive. Probably is in the data set but not clear where. | National level. |
Temporal | Information required for describing time-based characteristics of the data (e.g. the date of publication, a time stamp) | Changes over time, but slowly. Periodic updates. Good enough for basic model. | Dates of a number of variables, dates at which particular observations happened. Temporal information is recorded in the data dictionary and pertains to a number of the events. Deals with observation time and event time. Has spell information. | |
Contributors | People, organisations, agents, ... | Data aggregator is known. Have detailed, manual description of data sources. | Organisational identifiers? Hospitals, clinics, temporary treatment units? To what extent was this collected and is it contained in the data model. There is a field for evaluator, which gives some indication. Identification risks, but may not have been collected. | |
Process | Description of processes, workflows, transformations, ... | Importance of this depends on the research question. This dataset has detailed human readable account of the process and provenance of the data product. | Document that documents the process that went into compiling the data. Internal documentation. | |
Provenance | (Related to process) Descriptions of the process used to create/produce/transform/publish data | Covered in cell above. | ||
Vocabularies / lists / classifications | Enumerated lists of terms that may be applied to content being described. | No obvious incompatibilities; can be worked around because of data dictionary. | Uses standard CDISC domains. Customisations can be rolled back into the CDISC ontology. CDISC share references a number of vocabularies. | |
Resources | Objects being described or referenced. May include datasets, but also publications, software, code, other metadata, ... | Not applicable for use case. | Varies from organisational source. Often a relatively raw data dump. Compiled from pdf forms - these can be referenced. | |
Datasets | Specific descriptions of datasets as primary objects | Yes. | Jay will explore this. | |
Observation / Capture | Classes/objects that describe the processes by which data is created, generated, captured, transformed. (QUESTION: Is this the same as Provenance/Process?) | See above, process description. Dictionary has information about the devices used to make measurements. | ||
Data | The logical structure of the data being described - variables, units of measurement, concepts, sample units and populations, records, datum(s), cells. (May or may not be a subset of datasets) | Aggregate data set. Estimate of gridded population against time. Dimensionality to be identified. | Detailed data dictionary. Separate standard that relates to SDTM standards called ADAM. Is IDDO using this? | |
Storage | The physical representation of the data (files, formats, locations, ...) | Netcdf, GEOTiff, ASCII | Part of the CDISC package. Typically xml. CDISC standards has tools. Integrated with clinical systems. Generated directly by clinical systems. | |
Access | Who can access the data and how | "You are required to login to download data or maps. Click "LOGIN" to proceed to log in or to register. If you click "CANCEL", you may browse the page but you will still be required to login to download data or maps." Is there an API to access these data? Or is it just be selection and download? | Data access committee. External community can apply for access. Data providers can determine to what extent they wish data to be made available. What criteria are used? How does the data gatekeeper manage this? What restrictions are imposed? Is access information documented? | |
Administrative / core / ... | Foundational classes for use in building the specification - e.g. identification, versioning, primitives | CDISC standard. | ||
| Web compatibility? | Is it easy to use in a web environment? URLs, etc | Requires download. | Possibly in so far as CDISC is, but IDDO restricts access. | |
| Is the resource maintained and supported? | Yes. | Curated and maintained by IDDO. | ||
| Updating? | Dynamic or batch? | Periodic, every 3-4 years. | Periodic according to the extent that different organisations provide the data. | |
| Capacity for extensions? | Over the past few years CDISC has an ontology (CDISC Share) to connect data in different CDISC domains. There is a standard resource to help. |
Discussion of Disaster Risk Reduction
Good practice guide? Maturity model? Guidance on aggregation of data for reporting.
Possible recommendation of training materials such as those that Ernie Boyko prepared for national statistics offices, international household survey?
Value of the standards is increased by tools. Importance of crosswalks between tools and standards.
Scoping exercise for Fernando and Virginia: how is it best if the data is described.
Columns in the table point out what is needed.
What is missing for the Ebola model?
Overall outputs of the workshop: a workflow that assists Sendai reporting on
WHO guidelines - deadline 21-23 November.
What can Sendai learn from the SDMX experience and process?
Solution for Virginia: AM session to prepare the outline of a 'paper' on this.
- Recommendations around process.
- Outline of a 'system' that assists reporting.
- Discussion of definitional and data issues.
Lessons from IHSN. Data gathering. SDMX, statistical data. MDG indicators were described in SDMX.
CDISC
Complex in terms of time: deals with event data, spell data.
Link to the wikipedia page.
Jay will explore this and identify how CDISC has been stretched.
To add variables there are solutions through the ontology.
Continuation of Pilot Notes Day Two
Infectious Disease Data Spreadsheet: https://docs.google.com/spreadsheets/d/1twjmiu0_3bk_zgwJQI8y4Jk1601UjF3IhpqpA4dsb_Q/edit#gid=1931579469
Access issues. IDDO is not the data owner.
WHO data set. Ebola data and statistics: http://apps.who.int/gho/data/node.ebola-sitrep.quick-downloads?lang=en -
JSON file has an implicit scheme.
There does not appear to be a well-defined point of contact / author to find out about the structure and metadata.
The WHO data download is querying a database with a number of filters.
Global Health Observatory resources Data query API http://apps.who.int/gho/data/node.resources.api
GHO Metadata link: http://apps.who.int/gho/data/node.metadata
We can use the existing WHO queries as an approximation. The queries can be customised and made through the API.
Related XML Schema available at: http://converters.eionet.europa.eu/schemas/620 (broken!)
Default is XML. Likely to be the most reliable of the formats.
Query being used by the website: http://apps.who.int/gho/athena/xmart/EBOLA_MEASURE/CASES,DEATHS.xml?filter=COUNTRY:*;LOCATION:-;DATAPACKAGEID:2016-05-11;INDICATOR_TYPE:SITREP_CUMULATIVE;INDICATOR_TYPE:SITREP_CUMULATIVE_21_DAYS
http://apps.who.int/gho/athena/xmart/DATAPACKAGEID/2016-05-11?format=json&filter=COUNTRY:SLE
The ultimate goal for IDDO is being able to map a new outbreak.
Possible project: describe a couple of standards and the design of the platform that can harmonise the data in real time. Can we make recommendations for the data types such that they can be more effectively and rapidly integrated.
Imagining a future outbreak: is it feasible to imagine applying standards by the groups closely monitoring
Get the data that we have now, do a data quality analysis. Survey the people responsible for generating the data and get some input into how the data was gathered and entered. That would allow improvements to the collection of the data in the spreadsheet.
Link to the IDDO CDISC Data Dictionary: https://docs.google.com/spreadsheets/d/1gvJ1pPcDaqdeetRjtVT8RZ7ZINxhS4bn7v9B7YOeVrE/edit#gid=610474778
http://converters.eionet.europa.eu/schemas --> collection of schemas, although, curiously, ghodata. 'schema' only has an 'xsl' link.
Data Access at IDDO. Patient level information harder to obtain. If I want anonymised data, but with age and sex how long would that take?