Resumen de High performance computing of massive Astrometry and Photometry data from Gaia
Javier Bernardo Castañeda Pons
Gaia is an extremely ambitious astrometric space mission adopted within the scientific programme of the European Space Agency (ESA) in October 2000. It aims to measure with very high accuracy the positions, motions and parallaxes of a large number of stars and galactic objects, including also for almost all the objects information about their brightness, colour, radial velocity, orbits and astrophysical parameters. Gaia requires a demanding data processing system on both data volume and processing power. The treatment of the Gaia data has been designed as an iterative process between several systems each one solving different aspects of the data reduction system. In this thesis we have addressed the design and implementation of the Intermediate Data Updating (IDU) system. IDU is the instrument calibration and astrometric data processing system more demanding in data volume and processing power of the data processing system of the Gaia satellite data. Without this system, Gaia would not be able to provide the envisaged accuracies and its presence is fundamental to get the optimum convergence of the iterative process on which all the data processing of the spacecraft is based. The design and implementation of an efficient IDU system is not a simple task and a good knowledge of the Gaia mission is fundamental. The design and implementation of IDU is not only referring to the actual design and coding of the system but also to the management and scheduling of all the related development tasks, system tests and in addition the coordination of the teams contributing to this system. The developed system is very flexible and modular so it can be easily adapted and extended to cope with the changes on the operational processing requirements. In addition, the design and implementation of IDU presents a variety of interesting challenges; covering not only the purely scientific problems that appear in any data reduction but also the technical issues for the processing of the huge amount of data that Gaia is providing. The design has also been driven by the characteristics and restrictions of the execution environment and resources -- Marenostrum supercomputer hosted by the Barcelona Supercomputing Center (BSC) (Spain). Furthermore, we have developed several tools to make the handling of the data easier; including tailored data access routines, efficient data formats and an autonomous application in charge of handling and checking the correctness of all the input data entering and produced by IDU. Finally, we have been able to test and demonstrate how all the work done in the design and implementation of IDU is more than capable of dealing with the real Gaia data processing. We have basically executed two of the IDU tasks over the first ten months of routine operational Gaia data. This execution has been the very first cyclic data processing level run over real data so far. Executing IDU at Marenostrum over that amount of data for the first time has been a challenging task and from the results obtained we are confident that the system, we have designed and that constitutes the bulk of this thesis, is ready to cope with the Gaia data according to the requirements sets. Furthermore, the presented design provides a solid IDU system foundation for the challenging task of processing the Gaia data during the forthcoming years.
