Formats Supported by NEP

Open DICOM, PDB/mmCIF, GRIB2, GeoTIFF, FITS, NASA CDF, and PDS4 files through the standard netCDF API — no code changes required.

Compression Application Areas

Lossless Compression Filters

NEP adds two HDF5 filter plugins to netCDF-4 for lossless compression:

  • LZ4 – high-throughput compression ideal for speed-critical workflows
  • BZIP2 – high-ratio compression for long-term archives

Both are exposed through the standard nc_def_var_* API. HDF5 loads them transparently at runtime when HDF5_PLUGIN_PATH points to the NEP plugin directory.

Format Readers

NEP registers User-Defined Format (UDF) dispatch tables so existing netCDF programs can open and read these binary formats with nc_open() and nc_get_vara():

  • Medical Imaging – DICOM
  • Structural Biology – legacy PDB and PDBx/mmCIF
  • Planetary & Earth Science – NASA CDF, PDS4, FITS, GRIB2, GeoTIFF

How Each Format Maps to netCDF

DICOM

DICOM pixel data and metadata are exposed through the netCDF API for AI/ML pipelines and PACS/DICOMweb-integrated archives.

  • Images and metadata accessible as netCDF variables and attributes
  • Integrates with existing PACS and DICOMweb archives
  • Lossless compression for long-term storage
  • Feed standardized arrays directly into AI/ML training pipelines
PDB / PDBx mmCIF

Native netCDF access to predicted and experimental protein-structure data from RCSB PDB, AlphaFold DB, and the ESM Metagenomic Atlas.

  • Read legacy PDB and PDBx/mmCIF as ordinary netCDF datasets
  • Access atom coordinates, occupancy, and metadata through variables
  • Compress rapidly growing predicted-structure archives losslessly
  • Power structural-bioinformatics and protein-folding pipelines
GRIB2

The WMO standard for gridded data, decoded with NOAA's g2c library. Each GRIB2 product maps to a separate NC_FLOAT netCDF variable.

  • Shared y and x dimensions across products
  • Variable attributes record discipline, category, and parameter number
  • Missing grid points marked with _FillValue
  • Hyperslab reads supported via start/count
GeoTIFF

Reads standard TIFF and BigTIFF files through libgeotiff and libtiff. Image bands, rows, and columns map to netCDF dimensions.

  • Single-band raster maps to 2D [y, x]; multi-band to 3D [band, y, x]
  • Coordinate variables built from GeoTransform tags
  • CF-1.8 grid_mapping attributes for georeferencing
  • Read-only access through the netCDF API
NASA CDF

Uses the NASA CDF library to expose CDF zVariables and attributes as netCDF variables and attributes.

  • CDF zVariables map directly to netCDF variables
  • CDF data types translated to closest netCDF types
  • FILLVAL renamed to CF-standard _FillValue
  • Read-only; rVariables are not supported
FITS

Uses CFITSIO. Each Header Data Unit maps to a netCDF group; images and table columns become variables.

  • Primary HDU image maps to root-group variable named image
  • Table columns become variables named from TTYPEn
  • Header keywords preserved as group-level string attributes
  • Standard mappings: BUNIT to units, BLANK to _FillValue
NASA/ESA PDS4

Parses XML label files with libxml2 and reads associated binary or ASCII data files. Validated with Cassini, MESSENGER, MAVEN, Mars 2020, New Horizons, and other mission datasets.

  • Array objects become netCDF variables with dimensions from Axis_Array
  • Tables become variables over a record dimension
  • Identification_Area and Observation_Area metadata become global attributes
  • Read-only access

Application Areas

Each vertical below uses NEP to expose domain-specific data through the standard netCDF API, eliminating custom parsers and accelerating AI/ML workflows.

Medical Imaging AI/ML

DICOM-to-netCDF bridging for large-scale medical-imaging AI/ML data pipelines and PACS/DICOMweb-integrated archives.

  • Expose DICOM pixel data and metadata through the netCDF API
  • Integrate with existing PACS and DICOMweb archives
  • Apply lossless compression for long-term storage
  • Feed standardized arrays directly into AI/ML training pipelines
3D scatter plot of the 4HHB hemoglobin X-ray crystal structure atom coordinates
Protein Structure Databases

Native netCDF access to predicted and experimental protein-structure data from RCSB PDB, AlphaFold DB, and the ESM Metagenomic Atlas.

  • Read legacy PDB and PDBx/mmCIF as ordinary netCDF datasets
  • Access atom coordinates, occupancy, and metadata through variables
  • Compress rapidly growing predicted-structure archives losslessly
  • Power structural-bioinformatics and protein-folding pipelines
Planetary Science

Cross-mission analysis of PDS4, PDS3, FITS, and CDF planetary-science data through standard netCDF and OPeNDAP tooling.

  • Map grouped file structures directly to netCDF groups and variables
  • Analyze multi-mission data with the same tools used for Earth data
  • Preserve provenance and metadata through netCDF attributes
  • Enable cloud-friendly access and AI-ready data cubes

Visualization Examples

Sample figures produced directly from NEP UDF files using the standard netCDF4 Python API and Matplotlib. The plots below read PDB and mmCIF structures through NEP and render atom coordinates as 3D scatter plots.

3D scatter plot of the 1J7W haemoglobin mutant crystal structure atom coordinates
1J7W mmCIF Structure

3D scatter of ATOM and HETATM Cartesian coordinates from the 1J7W.cif deoxy haemoglobin beta-Y-Q mutant crystal structure, opened through the NetCDF mmCIF UDF interface.

3D scatter plot of the 4HHB hemoglobin X-ray crystal structure atom coordinates
4HHB Legacy PDB X-ray Structure

3D scatter of ATOM and HETATM Cartesian coordinates from the 4HHB.pdb hemoglobin X-ray crystal structure, opened through the NetCDF UDF interface.