How to access PAUS data (raw and reduced images)

Raw images, one directory per observation set: https://archive.pau.pic.es/raw/
Reduced images, one directory per production: https://archive.pau.pic.es/red/

How to access PAUS database (catalogs and metadata)

It is hosted on a PostgreSQL database at PIC. This is the information needed to connect:
  • hostname: db02.pau.pic.es
  • username: readonly
  • password: PAUsc1ence
  • database: dm

PAUS catalogs in CosmoHub

  1. PAUS Wide Deep Fields with BCNz
  2. PAUS+COSMOS photo-z catalog
  3. Flagship-PAU
  4. Galaxy Pairs identified in PAUS
  5. PAUS+COSMOS flux catalog
  6. PAUS CIGALE catalog W3
  7. Flagship-PAU

PAUdb connection instructions

Click here for a presentation of the basic concepts.

For just single queries, you can connect directly to ui.pic.es, query the db and save the result in a csv file in your ui home (you have 10 GB disk space available), and then copy the file to its final destination with scp.
Example query from an ui:
psql -U readonly -h db.pau.pic.es dm -c ‘SELECT * FROM table_name LIMIT 10’ -A -F ‘,’ > test.csv

If you have many queries to run, you need to create a tunnel: choose an arbitrary local_port (for example: 2000) and copy-paste the following line:

ssh -L {local_port}:db.pau.pic.es:5432 ui.pic.es

You have to use your PIC credentials to login and keep this terminal open.

From another terminal, you can query the db schema called “dm” (where all the production data is stored), with psql, or add the instructions for connection to your script. The password of the “readonly” user is given in the previous section.

psql example:

psql -U readonly -h localhost -p {puerto local} dm -c ‘SELECT * FROM {table_name} LIMIT 10’ -A -F ‘,’ > test.csv

python script example:

import sqlalchemy as sqla
import pandas as pd
dsn = ‘postgresql://readonly:{password}@localhost:{local_port}/dm’
engine = sqla.create_engine(dsn)
sql = “SELECT * FROM {table_name} LIMIT 10”
df = pd.read_sql(sql, engine)

PAUdb connection instructions from jupyter.pic.es

If you want to connect from jupyter.pic.es (updated in 10/05/2024):

You need a python environment that contains sqlalchemy and psycopg2-binary

You can open a terminal through your jupyterhub, activate your python environment and then install the packages:

pip install sqlalchemy
pip install psycopg2-binary

Then, in your notebook:

import sqlalchemy as sqla
import pandas as pd
dsn = ‘postgresql://readonly:PAUsc1ence@db.pau.pic.es/dm’
engine = sqla.create_engine(dsn)
sql = “SELECT * FROM star_zp LIMIT 10”
df = pd.read_sql(sql, engine)