File Types
==========

This page documents the file types that are used and/or generated by TXPipe. File types in TXPipe are all associated with a class that ceci uses to open, read, and write them. Some of the classes are generic, like ``HDFFile``, and some are specific like ``ShearCatalog``.

Ingested Catalog Files
----------------------


TXPipe starts from galaxy catalogs and various bits of anciliary information.  In most cases Ingestion stages generate these files from external data sets, such as GCRCatalogs, or generates mock data for them. Some are optional depending on the pipeline being run.

The catalog inputs are all in HDF5 format, and are:

- :ref:`Photometry Catalogs`: a general-purpose catalog of detected objects and their photometry.
- Shear Catalogs: galaxy shape/shear catalogs. TXPipe supports several different types:

   - :ref:`Metadetect catalogs`: from the method of `Sheldon et al <https://arxiv.org/abs/2303.03947>`_.
   - :ref:`Metacal catalogs`: from the method of `Sheldon & Huff <https://arxiv.org/abs/1702.02601>`_.
   - :ref:`Lensfit catalogs`: a catalog of galaxy shapes measured by Lensfit.
   - :ref:`HSC catalogs`: a catalog of galaxy shapes measured by ngmix.
   - :ref:`Generic shear catalogs`: a catalog of mock galaxy shears requiring no calibration
- :ref:`Star catalogs`: a catalog of stars, used for PSF-related diagnostics.
- :ref:`Exposure catalogs`: a catalog of exposures, used for more diagnostics.
- :ref:`Cluster catalogs`: a catalog of cluster locations and richnesses

The other inputs are:

- Depending on the algorithms chosen, possibly photo-z training or prior-related information.

Intermediate Catalog Files
--------------------------

Various stages in TXPipe generate intermediate catalogs, which are used as inputs to later stages. They are all saved in HDF5 format.

Those associated with photo-z are described in the next section. 


- :ref:`Tomography catalogs`: contain the tomographic bin index for each galaxy.

    - :ref:`Lens tomography catalogs`: additionally contains lens weights.
    - :ref:`Shear tomography catalogs`: additionally contains calibration and n_eff and sigma_e
- :ref:`Random Catalogs`: galaxies randomly distributed within the survey mask
- :ref:`Binned catalogs`: calibrated/weighted versions of the inputs catalogs, binned by tomography.

    - :ref:`Binned shear catalogs`
    - :ref:`Binned lens catalogs`
    - :ref:`Binned random catalogs`
    - :ref:`Binned star catalogs`

Photometric Redshift Files
--------------------------

- :ref:`Photo-z Files`: per-galaxy photo-z PDFs and ensemble (stacked) per-bin n(z) information.

    - :ref:`Photo-z PDF Files`
    - :ref:`Photo-z n(z) Files`

Map Files
---------

Various TXPipe stages generate maps of different kinds. They are stored in HDF5 files. An overview is here: :ref:`Map Files`, and individual map types are described here:

- :ref:`Mask`
- :ref:`Source Maps`
- :ref:`Auxiliary Source Maps`
- :ref:`Lens Maps`
- :ref:`Density Maps`
- :ref:`Auxiliary Lens Maps`
- :ref:`Convergence Maps`
- :ref:`LSS Weight Maps`
- :ref:`Source Noise Maps`
- :ref:`Lens Noise Maps`


Collected File details
----------------------

.. toctree::
    :maxdepth: 1
    :caption: Collected details on all file types

    file_details/binned
    file_details/clusters
    file_details/exposures
    file_details/generic_shear
    file_details/hsc
    file_details/lensfit
    file_details/maps
    file_details/metacal
    file_details/metadata
    file_details/metadetect
    file_details/photometry
    file_details/photoz
    file_details/randoms
    file_details/stars
    file_details/tomography