SDMX implementation process
Users of SDMX
SDMX was launched for the exchange of official statistics between international organisations, as well as between international organisations and their constituencies, such as the central banks and statistical agencies in their member countries.
For the European Statistical System (ESS), this has taken the form of exchanges between Eurostat and the national statistical offices of the EU members.
However, SDMX is also used by organisations outside the world of official statistics. It may be of interest to any organisation that collects, processes, analyses, and disseminates statistical data and metadata.
Roles in an SDMX project
Within any given organisation, there are generally 3 roles and teams in an SDMX project:
- initiators, represented by a business unit that has a business case for an SDMX project
- facilitators, represented by IT units that are involved with either developing tools or establishing the necessary IT architecture, as well as SDMX specialists who develop the relevant SDMX objects, such as data structure definitions (DSDs), metadata structure definitions (MSDs), code lists or concept schemes
- implementers, who provide the data by implementing the project to provide SDMX-compliant files
An SDMX project typically brings together statisticians, economists, methodologists, and experts in dissemination and information technology. This is why it is vital for people to 'speak the same language', measing to use shared standards and a shared vocabulary.
Implementation of SDMX
An SDMX project follows successive project management steps, meaning preparation, compliance, implementation, and production.
You can refer to this working checklist of the steps required in an SDMX project. A simplified abstraction of this is shown below, adapted for those involved in the ESS.
Preparation
The preparation phase of an SDMX project is critical. In this phase, the initiators and facilitators determine the project’s objectives, scope, expected benefits, and outputs. It is the moment to specify needs, plan, and organise.
Some of the key questions to be asked, among others, are:
- why do you want SDMX?
- what is the timetable?
- what risks are involved?
- what production systems, file formats, and code lists are currently in use?
- what is the frequency of data flows?
- who will be involved?
By the end of this phase, the goals of the SDMX project should be clear. So should the timetable for implementation, a draft project plan, and roles and responsibilities. The key decision is whether to go ahead with the project or not.
Compliance
The compliance phase is time-consuming. In this phase, the initiators and facilitators design the system and plan the sequence of the workflow.
Steps are taken to analyse the current exchanges, decide what can be reused, define the concepts, define the DSD matrix, and design supporting objects.
Implementation
The implementation phase brings together facilitators and implementers. At this building stage, SDMX objects, particularly the DSDs, are made available in something called the SDMX registry. The appropriate IT infrastructure is established, pilot projects are conducted (including testing and review), the last changes are made, the roll-out schedule is agreed upon, and support is provided.
SDMX registries are used by data providers and data collectors to store and access the objects needed for storing all the artefacts or products needed for those statistical information exchanges using SDMX.
These metadata repositories do not store the actual data or reference metadata. They are stored in the systems of the data providers. The registry only provides the information needed to access and interpret the content of the exchanged statistical data and metadata sets.
What kind of artefacts could a registry store? Among other items, a registry would store DSDs and their related artefacts, such as concept schemes, code lists, and data flows, as well as metadata structure definitions.
Access to a registry
Any SDMX registry is publicly and freely accessible to everyone. Advanced features can be enabled that allow subscribing to specific element or displaying linked structures and versioning of objects (SDMX objects change the version when they are updated). This implies ‘strong dependencies,’ which have a scheduling impact on tasks.
Potential users
An SDMX registry could be of interest to any group that exchanges statistical information in SDMX. Interested groups are typically international organisations and countries. The open-source Euro SDMX registry software is available for those exchange partners that are interested.
However, most people are interested in accessing the existing SDMX registries for using the SDMX artefacts that are contained within them.
Key registries
A number of registries exist, but there are 2 key ones that relate to Eurostat. The Euro SDMX registry and the SDMX global registry.
For artefacts related to exchanges of official statistics between EU members and Eurostat, the relevant registry is the Euro SDMX registry site.
In 2011, an SDMX Global Registry site was launched for artefacts related to exchanges of official statistics at a worldwide level. These are usually between international organisations or between some countries and their corresponding international organisation.
Production
The production phase is the final goal, when SDMX-compliant data and metadata can be used in exchanges. SDMX objects will continue to need regular maintenance, reflecting the need to be flexible to accommodate new coverage, new needs, new codes, etc.