Intra-EU trade - exchange of micro-data

This article presents the innovative statistical approach taken for modernising Intrastat, the European Union (EU) data collection system on trade in goods between Member States (intra-EU trade), which was introduced in January 1993 together with the Single Market.

The approach is to exchange micro-data on intra-EU exports between Member States allowing them to use those mirror data for the compiling of their own intra-EU imports statistics. This approach could reduce the administrative burden on Intrastat reporters on the imports side while maintaining data quality.

In order to test the technical and statistical feasibility of this innovative approach, the European Statistical System set up in 2012 a pilot project under the name of SIMSTAT (where 'SIMSTAT' stands for 'SIngle Market STATistics') which was conducted over 4 years with a successful outcome paving the way to a future European regulation.

Full article

Intrastat – system for collecting data on intra-EU trade in goods

The system for collecting data on intra-EU trade in goods is called Intrastat. It was introduced in January 1993, coinciding with the start of the Single Market, when customs declarations were no longer available as the source of data. Consequently, a new system was set up, with data collected directly from businesses in the Member States.

Figure 1: Number of Intrastat reporters, 2015

Intrastat is considered to be the largest business survey in the EU. In 2015 it was estimated that there were around 3.7 million European businesses involved in trade in goods between the Member States, out of which only about 13.1 % – around 485 000 – were liable to Intrastat reporting. Out of these 485 000 reporting businesses, 207 000 (43 %) reported only imports, 129 000 (27 %) only exports and 150 000 (31 %) reported both flows.

From the beginning of Intrastat, it became clear that the reporting burden imposed by Intrastat on businesses was quite heavy. The Intrastat system has undergone a number of changes and amendments of legislation in its 20-year history. These revisions have however not changed its basic characteristics but have rather enabled Member States to exempt more of their smallest businesses from Intrastat reporting obligations without making too drastic compromises in quality. While the objective of these revisions has always been clear, there has been recurring discussion on how the Intrastat system should be further simplified. Two alternative approaches were often put forward: raising the exemption thresholds and a single flow system.

Raising the exemption thresholds

Exempting more of the smallest traders from Intrastat reporting can be achieved by reducing the minimum coverage requirement in the Intrastat legislation. This makes it possible to raise the exemption thresholds. This is the approach which has been followed in previous Intrastat simplification rounds, leading to decrease the minimum coverage rates from the initial 99 % of total trade by value to the current 97 % for exports and 93 % for imports. There are, however, limits with this approach. Any decrease in the coverage rates (increase of the exemption threshold) necessitates estimation of a bigger (non-collected) volume of trade. For this reason there is a trade-off between the burden reduction and the targeted level of data quality.

Table 1 shows the exemption thresholds in individual Member States, broken down by imports and exports, as they were in force in 2015. The table also shows the coverage rates achieved by the Member States in 2015. When compared to the minimum coverage rate required by the legislation in 2015 (i.e. 97 % for exports and 93 % for imports) one can observe that 24 Member States collected more than the minimum required coverage rate for imports; the corresponding number of Member States for exports was 19. This suggests that most of the Member States prefer to collect more than they are obliged to do, in order to ensure high quality of data.

Table 1:Exemption thresholds and trade coverage, 2015

Single flow system

The current system of production of statistics on trade in goods between Member States is based on collection and compilation of both imports and exports in each Member State. In a single flow reporting system, statistics would be compiled by collecting data in each Member State for only one flow and deriving the other flow from the mirror statistics of the partner country. As there are fewer businesses exporting than importing, the single flow option based on collecting only exports could imply a significant burden reduction. The reporting on imports would become redundant and a large number of businesses could be exempted either completely (in cases where the given business has Intrastat reporting obligation for imports only) or partially (in cases where the given business has Intrastat reporting obligations for both exports and imports) from reporting obligations.

A single flow system in practice exists in the compilation of trade in goods between the United States and Canada. In this case, each country only collects import data and then exchanges them with the other country to obtain the country’s export data.

Despite significant burden reduction possibilities, the single flow system entails certain risks and shortcomings which outweigh the benefits. The most significant one is the loss of control over the imports statistics, as they will be fully determined by the exports of the partner countries. Moreover, because currently there are asymmetries in mirror trade flows between Member States, a switch to a single flow system could have a significant impact on the imports statistics and consequently on Member States’ trade balances and GDP.

The micro-data exchange approach

The very first proposal was put forward originally by Eurostat, in 2012 in response to a European Council's request to redevelop Intrastat system to reduce the administrative burden while at the same providing high quality statistics. This proposal aimed at introducing an innovative approach of compiling intra-EU trade statistics and reducing the administrative burden.

The European Statistical System Committee (ESSC) agreed on 24 May 2012 with the main principles of this new proposal and decided to launch a specific pilot project on micro-data exchange under the name of SIMSTAT pilot project.

The European Statistical Advisory Committee (ESAC) in its 12th meeting on 26 October 2012 also expressed a positive opinion on the approach saying that it "stands for a European approach for producing high quality international trade statistics. This is a change of paradigm in the ways European statistics are produced and disseminated".

After a successful piloting phase described in details in the following paragraph, the European Statistical System in May 2016 agreed on the key elements for a new approach to modernise the whole Intrastat system. The orientation taken builds upon the micro-data exchange approach:

Creation of an additional data source by making the exchange of micro-data on intra-EU exports among Member States compulsory

In contrast to the compulsory nature of the sharing of micro-data, Member States decide to what extent these micro-data are used together with other sources in the compilation of the imports statistics.

Simplification of the compilation requirements for Member States on the import side

This enables Member States to design the compilation of their import statistics according to their needs and opens up the possibility for Member States to be freed of any type of thresholds system, to use other data sources as well as innovative compilation methods.

Maintenance of the strong quality criteria of the statistical micro-data exchanged on the export side.

This approach adopts the principle that data collected and available within the European Statistical System (ESS) need not to be collected more than once. Thus, each transaction collected in one Member State may serve as a data source for two Member States: first, for compiling the intra-EU exports of the exporting Member State and, second, for compiling and/or verifying the intra-EU imports of the partner Member State.

As a consequence, the benefit of the approach is two-fold:

reduce the administrative burden on Intrastat reporters on the imports side and at the same time; and
guarantee that the statistical information provided is fit for the purpose and of high quality.

The SIMSTAT pilot project

The SIMSTAT pilot project on the exchange of micro-data on intra-EU trade in goods aimed to demonstrate the technical feasibility of the exchange of micro-data. The project involved both technical and statistical aspects. The technical aspects concerned the setting up of an IT system of exchange, testing of the system's functionalities, reliability, security, integrity and scalability as well as the network traffic and the management of big data volumes. The statistical aspects related to timeliness (including punctuality, availability of VIES data), accuracy (coverage, estimations, revisions, confidentiality, validation) and comparability of exchanged data with those collected nationally (values, quantities). Moreover, possible data coverage issues for the smaller Member States were also examined.

The implementation period of the SIMSTAT project is from June 2012 to May 2016. The period is divided in three main phases:

Phase 1: Feasibility study for the micro-data exchange (June 2012 – 2nd quarter 2013)
Phase 2: Development of the system for micro-data exchange (2nd quarter 2013 – 2nd quarter 2015)
Phase 3: Pilot testing of the system for micro-data exchange (2nd quarter 2015 – 2nd quarter 2016).

The first phase of the project was completed in June 2013. During this first phase a feasibility study was conducted and the decision was taken to pursue a technical solution based on centralised data Hub system using the Common Communication Network (CCN) as the transmission channel. In addition, certain agreements on the content, organisation and assessment criteria of the pilot micro-data exchange were reached and grant agreements (individual and multi-beneficiary – ESSnet) with the Member States were signed, enabling Member States to undertake the necessary preparations for the participation in the pilot exchange.

The second phase of the project finished in April 2015. The main tasks of the second phase were to develop the SIMSTAT information system and to ensure secure connections through the CCN network. Moreover, the organisation of the pilot exchange, the procedure to analyse exchanged data, to document the results and to draft the reports were agreed upon during the second phase.

During the third and final phase the exchange of micro-data between 20 Member States (Austria, Bulgaria, Croatia, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Italy, Latvia, Lithuania, Luxembourg, Malta, Poland, Portugal, Romania, Slovakia and Slovenia) took place. This was followed by a thorough analysis of the results and evaluation of the whole system.

The technical system for micro-data exchange

A regular and voluminous exchange of micro-data on intra-EU trade in goods between Member States during the data production cycle needs a new, sophisticated and high-performance IT solution due to its security and architectural dimensions. A feasibility study investigated various technical options, analysed the merits and drawbacks of each of them, evaluated the risks and benefits and assessed the implementation costs. On that basis, the study recommended the creation of a centralised Hub-based system as the most advantageous. In this system all the Member States involved in the SIMSTAT pilot project submitted data according to standard processes, formats and technologies from their systems to the data Hub; the data Hub split the received files by the partner Member States and forwarded these to them. Initially, Eurostat played the role of the data Hub.

The following diagram describes the system's high level architecture:

Figure 2: Micro-data exchange - High level architecture

In addition to the centralised data Hub solution, it was also recommended to use CCN network, owned by Directorate General TAXUD, as the transmission channel. This network is currently being used by the Member States for VAT and VIES data exchange between Customs and tax administrations. This recommendation was in line with the ESS.VIP Programme where the need of a modern network for the ESS was already recognized. The advantages of using CCN were evaluated and outnumbered those of using current Eurostat Edamis/Statel network. The facility of sending data between any participant in the network, the very sophisticated technology and the highly secure data exchange infrastructure are only some of the advantages.

Data security plays a crucial role in the micro-data exchange. The micro-data are, by definition, confidential. They have to be treated and transmitted securely in compliance with EU regulations. The Hub was hosted in a secure infrastructure segregated from the standard infrastructure of the European Commission, establishing secure connections with IT platforms and providing services allowing accredited users to interact with confidential data that they have permission to access. All data transmitted over the network were encrypted from end to end. Eurostat used additional encryption on the top of that offered by CCN.

Statistical aspects of micro-data exchange

Although efficient and secure IT solution is vital for the micro-data exchange, the success of the project also depended on statistical characteristics, like content of the data to be exchanged, identification of the importer, data quality, timetable for data exchange fitted to the production cycle of the Member States, confidentiality of data, etc. All these issues were analysed by Eurostat and the Member States and appropriate solutions were sought. The following paragraphs give an overview of these issues.

Content of the data to be exchanged

In the planned micro-data exchange system, the intra-EU exports data of a Member State will serve as a data source for the intra-EU imports data of the partner Member State. Only the availability of a commonly agreed list of harmonised variables makes it possible to ultimately replace the collection of imports data with exchanged exports data without compromising statistical quality.

While collecting data on exports, it was proposed that Member States additionally collect the unique identification numbers (VAT numbers) of the trading partner in the importing Member State. This, together with the idea of the exchange of these micro-data among Member States, is the backbone of the new approach. It will enable Member States compiling imports to use the exchanged exports micro-data knowing the national importer of the goods. This information is essential to monitor the population of businesses involved in imports to ensure that the results cover Member State's total imports. In addition, Member States compiling imports need to know who their national importers of the goods are, in order to integrate intra-EU trade in goods statistics with other business and economic statistics, in particular the National Accounts. This information is also important for data on Trade by Enterprise Characteristics (TEC).

Timeliness and integration into the production process

The timetable is one of the important prerequisites for a well-functioning micro-data exchange system. It has to be agreed and integrated into the production process in such a way that the current national obligation of transmitting data to Eurostat, to other main national users like National Accounts and Balance of Payments (BoP) statistics, and the national release calendar will not be affected.

Data validation and quality issues

Exchange of micro-data of high quality is crucial for the success of a modernised system making possible the use of data collected by other Member State. The Member States sending their micro-data to the partner countries have to ensure that they have carried out sound validity and consistency checks, and the exchanged datasets do not include any major errors. The Member States have agreed to a minimum set of data validation rules to be applied by all Member States sending micro-data in order to ensure the quality of data.

Confidentiality issues of micro-data exchange

As micro-data contain sensitive information on individual transactions, the Member States will have to agree on a common understanding about the treatment of the data received by them from other Member States in accordance with common confidentiality standards. The access to the micro-data will be limited only to the persons directly involved in collecting, processing, compiling and quality checking of Intrastat data in the Member States. The micro-data shall not be disseminated or further transmitted to other parties within or outside the Member States. For the SIMSTAT pilot project all information transmitted, exchanged or otherwise processed were used exclusively for analysing the feasibility of the IT system, assessing the usability of the data for eventual production of international trade in goods statistics, and for quality checks.

Organisation of the pilot exchange of micro-data

As mentioned above the SIMSTAT pilot project aimed to prove both the technical feasibility of the exchange of micro-data on intra-EU trade in goods and the feasibility of the use of the exchanged data as an additional data source for compiling intra-EU imports statistics.

The main objectives of the pilot exchange were formulated as follows:

The new IT infrastructure (common system, adapted national systems, network) for micro-data exchange is operational.
The security of the IT infrastructure (common system, network) corresponds to the requirements for exchange of confidential data.
The entire process of creating, transmitting, processing and reusing micro-data proves operational and does not involve structural issues.
The timeliness meets the existing legal deadlines for data transmission to Eurostat and the requirements of main data users.
The exchanged micro-data could be used as a substitute for the collected imports data.

The exchange of micro-data was realised through a new IT system which was delivered in the 1st quarter of 2015 and tested through pilot exchange by 20 Member States during the 2nd and 3rd quarters of 2015.

When evaluating the SIMSTAT project, it was important to differentiate between the process of exchanging the micro-data and the product of this exchange, i.e. the exchanged data. The process consisted of series of steps to transfer the selected set of micro-data on intra-EU exports from the sender's environment to the receiver's environment via the central node (the data Hub), using the CCN network. The correct functioning of the Member States' SIMSTAT application, the proper transmission of micro-data via the CCN network to the data Hub and from the data Hub and the efficient functioning of the data Hub were assessed. In addition, the data exchanged were assessed in terms of their quality, comparability and re-usability.

Each participant in the pilot exchange gave written feedback on the above-mentioned elements. The consolidated report on the pilot exchange is the main output of the pilot exchange and can be found at the following link.

Towards an EU regulation and national implementations

The translation of the micro-data exchange (MDE) approach into a real statistical production system will now require considerable development efforts for the whole ESS. The European Commission has put forward a European regulation proposal in order to modernise the whole legislative framework on the intra-EU trade in goods statistics and among other things, make the exchange of intra EU trade in goods micro-data mandatory.

It is, however, worth taking note of the following issues.

MDE is not single flow.

It is up to the Member States to decide to what extent and how they will make use of the additional source of data made available to them. Member States can replace their collection of imports data completely or partially, or can continue collecting imports data. The future legislation governing international trade in goods statistics will continue to require Member States to compile and transmit statistics covering both flows.

Identification number of the partner in Member State of import

One central issue is the collection of the identification number of the partner in the Member State of import. The future legislation will require the mandatory exchange of this variable.

Data quality

It has to be acknowledged that current Intrastat data are not without deficiencies in quality. One indication is the existence of significant disparities between mirror data of Member States, especially at detailed level. These asymmetries impact the comparability and coherence of GDP and BoP figures. MDE has the potential to improve the intra-EU trade in goods data quality. Exchanged micro-data will better allow data compilers to compare the collected micro-data with the mirror data received from other Member States, identify any discrepancies, try to find reasons for these and gradually reduce them by making the way data is treated and compiled more consistent among Member States. Thanks to MDE, the mirror comparison will be possible on a regular and timely basis during the normal production process (currently mirror exercises are run ad-hoc and with a delay of 1-2 years).

Consequence for users

The new approach will not bring any change in the main features of the output of intra-EU trade in goods statistics. No change in the product or country breakdown, product detail (CN8 level) nor frequency or timeliness is planned. It will only improve the comparability of the published statistics as explained above.

Consequence for data producers

The SIMSTAT pilot exchange entailed investment by the competent authorities of Member States. The Commission (Eurostat) supported these efforts through grants for methodological work, quality analysis, and assessment of impacts and adaptation of the IT infrastructure. The implementation of the full MDE approach, if agreed, will require further effort for making available good quality micro-data on time according to a predefined calendar and analysing these micro-data as a new data source and observing the quality of final data. These efforts will be rewarded by the opportunity to simplify the reporting obligations of importing businesses and to introduce new quality controls that will reduce the asymmetries present in current Intrastat data. The added value of MDE will grow in the mid-term to long-term as the competent authorities of Member States develop better knowledge of the micro-data and skills for their re-use.

Consequence for data providers

The new approach could enable more businesses to be exempted from statistical reporting. Since there are fewer companies exporting commodities than importing them, collecting data on exports represents a lesser burden on businesses than collecting data on imports. The possible removal of reporting obligations for importers would relieve the European businesses even though the variable on importer identification number (partner ID number) is added to the export survey of Member States. A wide scale collection of data on imports might not be needed if information can be derived from the "mirror" export transactions already recorded by the partner Member State in the trade operation.

Conclusion

The positive outcomes from the SIMSTAT pilot project on micro-data exchange run between 2012-2015 were used by the European Statistical System Committee as a proof of concept for a secure and sustainable infrastructure that provides key capabilities for information exchange between the different partners in the ESS, including micro-data.

In its meeting of May 2016, the European Statistical System Committee decided to put in place a modernised system for the compilation of intra-EU trade in goods statistics. The coming years will thus now be dedicated to the practical implementation of the MDE approach into a real statistical production system.

In order to meet this challenge, the European Statistical System set up a wide scale implementation project called "Modernisation of Intrastat".

In addition, the European Commission put forward a proposal to establish a common legal framework for the development, production and dissemination of European statistics related to business structure, economic activities and performance, as well as on international transactions and research and development activities in the EU economy; and for the European network of national statistical business registers and the EuroGroups Register. This proposed new regulation will, among other things, include the exchange of micro data for the purpose of intra-EU trade in goods statistics.

Context

International trade in goods statistics provide very detailed information on the values and quantities of goods traded between countries. They are an important multi-purpose source of information for numerous public and private sector decision makers. However, the system for collecting data on intra-EU trade in goods (Intrastat) is a domain with high reporting burden on businesses. That is why the Council has repeatedly called for ways of reducing the burden.

The Council, in its November 2011 meeting, called upon "ESS to take the necessary steps in the area of international trade statistics to address current and future user needs and to take effective measures ensuring a substantial reduction of the response burden by redeveloping Intrastat (not excluding the option of a "single flow system"), while maintaining a sound level of quality needed for, e.g., European System of Accounts purposes". This request demanded a clear reply from the ESS.

As a result, an innovative proposal based on micro data exchange was put forward in 2012 in response to the Council’s request. Since the traditional simplification efforts only brought limited results not sufficient to achieve the targeted reduction, the Commission conceived this proposal to deliver a new business architecture for Intrastat.