TXPipe File Tags

TXPipe saves everything it does in files saved in the output_dir directory specified in the pipeline YAML file. This page is useful if you want to learn what one of those files means and find a link to a page that describes the format of that file.

TXPipe files are organized by tags, which are usually just the stem of the file name (i.e. without the directory or suffix). The tags input and output by each staged are defined in the stage class. Each file has a specific type

This page briefly describes all the file tags that TXPipe currently generates and uses, and links to pages describing the format of each type of file.

Aliases

Any stage can be “aliased” to a different tag when the pipeline is defined in the pipeline YAML file. This means that the name used for the stage internally can be different from the name used in the file names. This is useful when there are pipeline stages that can be run multiple times on different data sets.

For example, the TXPhotozPlot class can be run twice, once on the source sample and once on the lens sample, to generate n(z) plots of each.

HDF5 Files

HDF files are a general-purpose file format for storing tabular data. They are the primary storage format for larger data sets in TXPipe, primarily because they are easy to read and write in parallel. Different groups of HDF5 files are described below.

Catalog HDF5 Files

TXPipe catalog files are almost all stored in HDF5 format. They are described in full on the File Types page.

HDF5 Files

File Tag	Kind	Description
binned_lens_catalog	Binned lens catalogs	The foreground lens (clustering) sample, split into tomographic bins, with weights
binned_lens_catalog_unweighted	Binned lens catalogs	The foreground lens (clustering) sample, split into tomographic bins, with only unit weights
binned_random_catalog	Binned random catalogs	The random catalog, split into tomographic bins
binned_random_catalog_sub	Binned random catalogs	A smaller sub-sample of the random catalog, split into tomographic bins. Used when the full large catalog is not needed and too slow.
binned_shear_catalog	Binned shear catalogs	The shear/source/lensing sample, split into tomographic bins and calibrated
binned_star_catalog	Binned star catalogs	The star catalog split into sub-classes, currently by brightness
cluster_catalog	Cluster Catalogs	Locations, redshifts, and richness of clusters
cluster_shear_catalogs	Cluster Shear Catalogs	An catalog of shear values around clusters.
exposures	Exposure catalogs	Catalogs centers of exposurs for use in systematics tests
lens_catalog	HDF File	A catalog of objects to be used as lenses (when something external is used instead of the photometry catalog
lens_tomography_catalog	Lens tomography catalogs	Tomographic selection information for the lens sample
lens_tomography_catalog_unweighted	Lens tomography catalogs	Tomographic selection information for the lens sample, without weights
photometry_catalog	Photometry Catalogs	Photometric measurements from which the lens sample is chosen
random_cats	Random Catalogs	A catalog of random objects following the same tomographic and location selection as the lens sample but with no underlying structure
shear_catalog	Shear Catalogs	Shear catalogs of various different types
shear_catalog_quantiles	HDF File	Measurements of quantiles of shear catalog columns such as SNR, size, etc.
shear_tomography_catalog	Shear tomography catalogs	Tomographic selection and shear calibration information for the source sample
spectroscopic_catalog	HDF File	Training set for the photo-z model
star_catalog	Star catalogs	Catalog of star locations and magnitudes for systematics tests

Map HDF Files

TXPipe map files are described in full under Maps Files.

HDF5 Files

File Tag	Kind	Description
aux_lens_maps	Auxiliary Lens Maps	Maps related to the lens sample, most notably the depth
aux_source_maps	Auxiliary Source Maps	Maps related to the source sample, such as PSF and weight
convergence_maps	Convergence Maps	Reconstructed convergence maps, typically starting from the shear maps
density_maps	Density maps	Over-density maps generated from lens number count maps
input_lss_weight_maps	Maps Files	Weight maps used in GLASS simulations
lens_maps	Lens maps	Weighted and raw number density maps of the source sample
lens_noise_maps	Lens Noise Maps	Density and number count maps for random halves of the lens and density maps
lss_weight_maps	LSS Weight Maps	Maps of weights for the lens sample
mask	Mask	Binary or fractional pixel coverage masks
source_maps	Source maps	Tomographic cosmic shear maps
source_noise_maps	Source Noise Maps	Tomographic cosmic shear map realizations with all object shears radomly rotated

Photo-z HDF Files

The Photo-z files page describes the types and formats of photometric redshift files in more detail.

HDF5 Files

File Tag	Kind	Description
lens_photoz_pdfs	Photo-z PDF Files	Per-object PDFs for the lens sample
lens_photoz_realizations	Photo-z n(z) Files	Per-tomographic bin photo-z realizations for the lens sample
lens_photoz_stack	Photo-z n(z) Files	Mean tomographic bin photo-z for the lens sample
shear_photoz_stack	Photo-z n(z) Files	Mean tomographic bin photo-z for the source sample
source_photoz_pdfs	Photo-z PDF Files	Per-object PDFs for the source sample
source_photoz_realizations	Photo-z n(z) Files	Per-tomographic bin photo-z realizations for the source sample

Miscellaneous HDF Files

Various miscellaneous HDF5 files with no common structure are also generated in the pipeline. See the generic HDF5 page for information on reading them.

HDF5 Files

File Tag	Description
brighter_fatter_data	Measurements of PSF size and ellipticity mismatch as a function of magnitude
density_shells	Simulation density shell maps when simulating log-normal maps with GLASS
glass_cl_binned	Tomographic log-normal C_ell realizations from GLASS
glass_cl_shells	Shell log-normal C_ell realizations from GLASS
response_model	A model used for generating mock shear catalog calibration distributions.
rowe_stats	Tabulation of the Rowe PSF statistics
star_density_stats	Cross-correlation measurements between star and galaxy positions
star_shear_stats	Cross-correlation measurements between star PSFs and galaxy shears
tau_stats	Measurements of PSF Tau statistics
tracer_metadata	Collected lens and source sample metadata. See Metadata.

SACC Files

SACC is a DESC library for storing cosmological measurements and all the metadata associated with them needed to perform parameter estimation. The TXPipe intefface class for SACC files is txpipe.data_types.SaccFile, but you can also just use the SACC library directly.

SACC Files

File Tag	Description
aperture_mass_data	Mass aperture statistic measurements
gammat_bright_stars	Diagnostic measurements of the tangential shear around bright stars
gammat_dim_stars	Diagnostic measurements of the tangential shear around bright stars
gammat_field_center	Diagnostic measurements of the tangential shear around field centers
gammat_randoms	Diagnostic measurements of the tangential shear around random positions
summary_statistics_fourier	3x2pt C_ell spectrum measurements with covariance, suitable for parameter estimation (may be blinded)
summary_statistics_real	3x2pt correlation function measurements with covariance, suitable for parameter estimation (may be blinded)
twopoint_data_fourier	3x2pt C_ell spectrum measurements with shot noise only (may be blinded)
twopoint_data_real	3x2pt correlation function measurements with shot noise only (may be blinded)
twopoint_data_real_raw	Unblind 3x2pt correlation function measurements
twopoint_gamma_x	Diagnostic measurement of cross-shear around lens sample
twopoint_theory_fourier	Theory prediction for the 3x2pt C_ell spectrum based on fiducial cosmology
twopoint_theory_real	Theory prediction for the 3x2pt correlation function based on fiducial cosmology

PNG Images

Images are used for both quality diagnostics and plots of measured summary statistics. These would mostly require tweaking to be publication-ready.

PNG Images

File Tag	Description
T_frac_psf_residual_hist	Histogram of fractional error in PSF size
bright_object_map	Map of distribution of bright object counts
brighter_fatter_plot	Brighter-fatter effect diagnostic plots
convergence_map	Map plot of the reconstructed convergence
density_cl	C_ell power spectrum of the density field
density_xi	C_ell power spectrum of lensing
density_xi_ratio	Ratio of C_ell lensing power spectrum to fiducial theory prediction
depth_map	Map of estimated n (default 5) sigma depth in a selected band
e1_psf_residual_hist	Histogram of PSF residuals in e1
e2_psf_residual_hist	Histogram of PSF residuals in e2
flag_map	Map of counts of flagged objects
g1_hist	Histogram of shear g1
g2_hist	Histogram of shear g2
g_T	Trend of shear as a function of quadratic size T
g_colormag	Trend of shear as a function of colors and magnitudes
g_psf_T	Trend of shear as a function of psf size
g_psf_g	Trend of shear as a function of psf ellipticity
g_snr	Trend of shear as a function of galaxy signal-to-noise
gammat_bright_stars_plot	Tangential shear around bright stars
gammat_dim_stars_plot	Tangential shear around dim stars
gammat_field_center_plot	Tangential shear around field centers
gammat_randoms_plot	Tangential shear around random positions
jk	Jack-knife regions used in shot noise estimation and 2pt measurement
lens_mag_hist	Histogram of lens magnitudes
lens_map	Map of lens sample counts
lens_nz	Lens sample ensemble photometric redshift n(z)
lens_photoz_realizations_plot	Realizations of lens sample photo-z
lens_snr_hist	Histogram of lens sample signal-to-noise
mask_map	Map image of mask values
nz_lens	Lens sample ensemble photometric redshift n(z)
nz_source	Source sample ensemble photometric redshift n(z)
psf_map	Map of the mean PSF ellipticity
response_hist	Histogram of the metadetection response values
rowe0	First Rowe statistic zero measurement plot
rowe134	Second set of Rowe stat measurement plots
rowe25	Third set of Rowe stat measurements plots
shearDensity_cl	Galaxy Galaxy-Lensing measured spectrum plot
shearDensity_xi	Galaxy Galaxy-Lensing measured correlation plot
shearDensity_xi_ratio	Galaxy Galaxy-Lensing measured correlation ratio to theory plot
shearDensity_xi_x	Galaxy Galaxy-Lensing cross-shear diagnostic plot
shear_cl_ee	Measured power spectrum E-mode plot
shear_cl_ee_ratio	Measured power spectrum E-mode ratio to theory plot
shear_map	g1 and g2 maps
shear_xi_minus	Shear correlation function plot
shear_xi_minus_ratio	Shear correlation function ratio plot
shear_xi_plus	Shear correlation function plot
shear_xi_plus_ratio	Shear correlation function ratio plot
source_mag_hist	Histogram of magnitudes of source sample
source_nz	Plot of source sample ensemble photometric redshift n(z)
source_photoz_realizations_plot	Plot of realizations of the source sample ensemble photometric redshift n(z)
source_snr_hist	Histogram of source sample signal-to-noise
star_density_test	Galaxy-star density correlation function plot
star_shear_test	Galaxy-star shear correlation function plot
star_star_test	Star-star shear correlation function plot
tau0	First PSF Tau statistic plot
tau2	Second PSF Tau statistic plot
tau5	Third PSF Tau statistic plot

Text Files

Smaller data items in TXPipe are sometimes stored in simple text formats.

PNG Images

File Tag	Description
calibration_table	A secondary training set for tomographic binning
g_T_out	Measurements of shear as a function of quadratic size T
g_psf_T_out	Measurements of shear as a function of PSF size
g_psf_g_out	Measurements of shear as a function of PSF shear
g_snr_out	Measurements of shear as a function of signal-to-noise
mock_shear_catalog	A mock shear catalog for testing
patch_centers	Locations of jackknife patch centres
rlens_measurement	Tangential shear in comoving coordinates

YAML Files

The YAML format is used for somewhat more structured key/value type data instead of plain text files.

PNG Images

File Tag	Description
fiducial_cosmology	A representation of the fiducial cosmology loaded by CCL. Has its own FiducialCosmology class.
star_psf_stats	PSF diagnostics for stars
tracer_metadata_yml	Collected metadata for the lens and source samples

Pickle Files

In a few miscellaneous cases, TXPipe uses the Python pickle format for storing data. We are generally trying to phase this out as it is not very interpretable, but RAIL makes extensive use of this for storing trained PZ models, because they are complex and varied.

PNG Images

File Tag	Description
flow	Example model for simulating mock spectroscopic population
lens_direct_calibration_model	Model for the NZDIR photo-z n(z) estimation for the lens sample
lens_photoz_model	Model for BPZ photo-z estimation PDF p(z) estimation
source_direct_calibration_model	Model for the NZDIR photo-z n(z) estimation for the source sample

Other Files and Directories

• lss_weight_summary - Directory

• map_systematic_correlations - Directory

• redmagic_catalog - FITS file

• ideal_specz_catalog - parquet file

• specz_catalog_pq - parquet file