Archive:European business statistics manual - data exchange - EDAMIS

This Statistics Explained article is outdated and has been archived - for updated information please see the dynamic version of the European Business Statistics Manual at: European Business Statistics Manual

A static full version of the European Business Statistics Manual was published in February 2021: European Business Statistics Manual — 2021 edition This article describes the system transmitting data from national data compilers to Eurostat — EDAMIS.

The format of the data exchange is described in a separate article, Data exchange in business statistics (part 2): SDMX.

The article is part of the online European Business Statistics Manual, a comprehensive guide to methodologies and how business statistics are produced within the European Statistical System (ESS).

Full article

Introduction

Eurostat is required to:

receive data from EU countries, mainly from national statistical institutes, but also from other sources
send data to EU countries
share data with third parties such as the European Central Bank (ECB), the OECD and outside contractors.

Many of the data are confidential, whilst some come in the form of large files.

In the past, data were sent in a variety of ad hoc ways, ranging from email (the commonest channel) to CD-ROMs. In 1996 it was decided to set up a Single Entry Point (SEP) for data transmission, so as to share data in a more structured, consistent and verifiable way.

The Single Entry Point was originally introduced under the names STADIUM (for data transmission) and EDIFLOW (for managing the dataset inventory). STADIUM consisted of a central server in Eurostat and individual client applications installed in national statistical institutes. An internet-based version of the client application was subsequently introduced so non-NSIs could use the Single Entry Point. At this stage, the server parts of STADIUM, EDIFLOW and the Internet client were combined and the data transmission system as a whole was renamed EDAMIS (ELECTRONIC DATA FILES ADMINISTRATION AND MANAGEMENT INFORMATION SYSTEM).

After Eurostat, together with EU countries, had been developing the Single Entry Point for a number of years, the meeting of Eurostat Directors of 12 February 2008 decided it would be compulsory within Eurostat to use EDAMIS to transmit all datasets from 1 July 2008, thereby establishing the mandate for a Single Entry Point. Subsequently, on 5 May 2008, Eurostat’s Director-General sent official letters to inform members of the Statistical Programme Committee (SPC) and Eurostat heads of unit of the decision. In May 2009, a report on the implementation of the Single Entry Point policy was presented to the European Statistical System Committee (ESSC).

Inventory

The dataset inventory within EDAMIS lists the statistical domains and the datasets within each domain. Dataset names are constructed by joining together:

the domain
the specific table/collection
the intervals at which data are collected.

Additional metadata concerning each domain and dataset are stored, including information about the unit and individuals responsible within Eurostat, the basis for the collection (such as a legal act) and the deadline for sending the data.

EDAMIS also holds a list of all organisations sending data to Eurostat, categorised as:

national statistical institutes
other national authorities (as referred to in Art. 5 of Regulation No 223 (ONAs)
other organisations or entity (OOE)

and information concerning the authorised senders in each of the organisations.

Currently, the EDAMIS inventory lists more than 200 domains and 1 500 datasets. Excluding metadata, there are more or less 1 200 datasets, collected mainly annually. The others are collected on a quarterly and monthly basis, while the rest are collected at intervals exceeding one year or are not collected at set intervals. Around 1 000 datasets use the EDAMIS ‘send data’ facility, while there are more than 150 webforms defined. More than 600 organisations are recorded as sending data to Eurostat. There are around 40 000 country/organisation/dataset links, with more than 23 000 to EU-28 national statistical institutes. There are almost 5 000 non-Eurostat persons registered in the EDAMIS system, including about 2 600 regular users.

Transmission tools

The Single Entry Point comprises:

the central server and inventory installed in Eurostat
4 client applications - EWA, EWP, EWF, EWS.

The EDAMIS WEB APPLICATION (EWA) is a client installed in the national statistical institutes. The application holds a local copy of the dataset inventory and users applicable to a particular national statistical institute. Transmissions between the application and the Eurostat server are by way of HTTP, through a telecommunications module known as STATEL.

The EDAMIS WEB PORTAL (EWP) provides the same functionality as the EDAMIS Web Applications, but is internet-based, which means it does not have to be installed locally by users. The EWP, which uses the EU Login user authentication system, is used primarily by smaller data providers (other national authorities and other organisations) but can also be used by people in national statistical institutions (NSIs) as the local EDAMIS Web Applications are synchronised with the EDAMIS Web Application and therefore the use of each is interchangeable. Data providers can use both EWA and EWP to send a regular data file of any format and size.

The EDAMIS WEB SERVICES (EWS) are mostly used for sending feedback to the users.

Finally, EDAMIS WEBFORMS (EWF) form part of the EWP and provide a fixed template for submitting data.

The EDAMIS system allows the user to select the ENCRYPTION of confidential data ‘from statistician to statistician’ (In any case, the transmissions of the files via the different tools are always encrypted). For standardised data transmissions, the EDAMIS system can make a VALIDATION of the file structure against a published DSD.

When the data files or webforms reach the central server in Eurostat, EDAMIS sends an official email acknowledgement (receipt) for the dataset/period to the actual data sender and any other individuals nominated by the sending organisation. The file is then delivered to a server/directory assigned to the relevant Eurostat production unit, and the individuals responsible for handling the data are notified by email that the data have arrived. Delivered files are named in a standard format in accordance with the DATASET NAMING CONVENTION (DSNC): the dataset name, the sending country, the period of the data (e.g. year/month or sequential) and the version. The IT system logs the whole transmission chain.

After the Eurostat production unit has processed the file, EDAMIS can, if appropriate, send the data provider FEEDBACK. This may take the form of written comments or log files showing any errors discovered in the course of file processing (e.g. when validated in EDIT). EDAMIS allows incoming feedback to be automatically FORWARDED to Eurostat contractors or other organisations with an interest, such as the OECD and the European Central Bank. In addition, Eurostat can transmit files to outside organisations and log data submissions made through EDIT. EDAMIS can also be used to send REMINDERS that data are expected, based on the intervals between data transmissions and the deadline for each dataset. Reminders can be sent before the deadline, on the deadline and after the deadline. They are sent if the obligation to send is laid down in a legal act or agreement (not if data submission is voluntary) and the dataset/deadline has not been sent by the date of the reminder.

Approximately 62 000 first instance transmissions are expected in 2017. 75 % of transmission deadlines are just 12 days, the busiest day of the year being 29 September.

Support

The first contact point for help with data transmission and data transmission tools is the Eurostat Data Transmission Helpdesk — EDAMIS SUPPORT. The Helpdesk also:

manages the dataset inventory
manages the dataset/country/organisation links
prepares and distributes a range of documentation about the tools available
maintains the EDAMIS Help Centre.

Through the application, the EDAMIS Support produces a compliance indicator report. The latter is the indicator of the extent to which each organisation has met its data transmission obligation (first instances of a dataset/period expected during the period under review, the EU-28 NSI average for 2016 has been over 90 %).

Eurostat has established a TRANSMISSION COORDINATOR NETWORK. Each national statistical institute elects a Transmission Coordinator (TCO), who is the main responsible person for contact with Eurostat, ensuring effective implementation of the electronic data exchange, and also the contact for members of their organisation. Most coordinators also support and assist members of other organisations providing data in their country. They meet annually in Luxembourg in a specific user group. This group supports the work of the relevant IT Working Group in the ESS. Both the Transmission Coordinators and EDAMIS Support manage EDAMIS users, using specific functions of the EDAMIS Web Portal.

The EDAMIS Web Portal offers a full range of REPORTING functions. Detailed information on domains, datasets, statistical products and organisations is available. Country/organisation links to domains and datasets can be produced, as can lists of people connected with these links. In addition to statistics and the Compliance Indicator for any required period, reports on data transmissions and timetables can be drawn up. The reports are available to all users registered with their national associations.

Contacts

EDAMIS Help Centre: http://webgate.ec.europa.eu/edamis/helpcenter/website/index.htm

Transmission Coordinators: http://webgate.ec.europa.eu/edamis/helpcenter/website/contacts/2011-11-10-Local-Coordinators.gif

Eurostat Data Transmission Helpdesk: ESTAT-SUPPORT-EDAMIS@ec.europa.eu