A framework for the publication of scientific databases
Query interface from Gaia@AIP
Today, the publication of research data plays an important role in astronomy and astrophysics. On the one hand, dedicated surveys like SDSS and RAVE, data intensive instruments like LOFAR, or massive simulations like Millennium and MultiDark are initially planned to release their data for the community. On the other hand, more traditionally oriented research projects strive to publish their data as a key requirement demanded by the funding agencies.
The common approach is to publish this data via dedicated web sites. This includes rather simple HTML forms as well as complex query systems such as SDSS-CAS. Most of these web sites are tailor made for the particular case and are therefore not easily transferable to future projects.
At Leibniz-Institute for Astrophysics Potsdam (AIP), we gained experience with both the maintenance and the development of such applications. It became, however, apparent that already the current plethora of applications constitutes a major challenge for maintenance expenses and scalability. In order to address these issues, we developed the Daiquiri framework, which is particularly designed to allow for different highly customizable web applications based on a common easily maintainable code base.
Daiquiri enables collaboration and institutions to create customized websites, comprising of the following features:
- An interactive Query interface enabling users to perform SQL/ADQL queries against catalog databases. The queries are analyzed using the queryparser and permissions are checked depending on user accounts and groups.
- Asynchronous database queries, which can take minutes or even hours.
- Download of the query results in different formats and visualization of the data.
- A programmatic interface to the database implementing the IVOA TAP protocol.
- A cone search API based on the IVOA Simple Cone Search recommendation.
- An integration into to IVOA registry of registry to make the VO endpoints available in applications like, e.g. topcat.
- A metadata management backend containing information about the database schemas and tables including DOI and UCD.
- The download of files connected to the database tables, including access restrictions.
- An OAI-PMH2 endpoint to make the metadata stored in the system available to harvesters.
- A cut-out API for multi-dimensional data (e.g. data cubes).
- A sophisticated user management system with customizable registration and confirmation workflows.
- A contact form connected to the management backend.
Daiquiri is based on Django and is written in Python. The following requirements are mandatory:
- Python
>=3.9 - PostgreSQL
>=14, MySQL>=5.6 - RabbitMQ
>=3.5(for asyncronous tasks like the query queue)
For demonstration, development or testing purposes, Daiquiri can be installed on Linux, macOS, or even Windows. If you, however, intent to set up a production enviroment, serving Daiquiri over a Network or the Internet, we strongly suggest that you use a recent Linux distribution, namely:
- Debian 11
- Ubuntu 22.04
Daiquiri is currently used on several sites hosted and maintained by the Leibniz-Institute for Astrophysics Potsdam (AIP):
- Gaia@AIP Services
- APPLAUSE archives
- MUSE-Wide survey
- GREGOR project and archive
- CLUES – Constrained Local UniversE Simulations project
- CosmoSim database
- RAVE Survey
You can find the detailed instractions for the installation of Daiquiri in the main documentation.
Main documenation: django-daiquiri.github.io