references/spectroscopy_analytical_formats.md
9.3 KB查看排版效果| 1 | # Spectroscopy and Analytical Chemistry Formats |
| 2 | |
| 3 | **Reviewed:** 2026-09-30 |
| 4 | **Executable scope:** No spectroscopy-native parser is bundled. General |
| 5 | CSV/TSV/JSON/NumPy/HDF5 inspectors apply only when a file is truly one of those |
| 6 | registered formats and do not add spectroscopy semantics. |
| 7 | |
| 8 | ## Capability boundary |
| 9 | |
| 10 | | Format | Bundled native inspection | Status | |
| 11 | |---|---|---| |
| 12 | | mzML/mzXML, MGF | No | Reference-only MS tooling | |
| 13 | | JCAMP-DX (`.jdx`, `.dx`) | No | Reference-only technique/version-aware parser | |
| 14 | | SPC and vendor spectroscopy binaries | No | Reference-only producer-specific parser | |
| 15 | | Vendor `.raw`, `.d`, `.fid`, `.dat`, `.out` | No | Ambiguous suffix/path; producer and format must be confirmed | |
| 16 | | CSV/TSV exports | General tabular scripts | Bounded aggregates only after delimiter, units, axes, and missing codes are confirmed | |
| 17 | | NPY/NPZ/HDF5 exports | General container scripts | Structural/bounded numeric inspection only; no instrument semantics | |
| 18 | |
| 19 | Unknown formats fail closed. Directory-based acquisitions are rejected by the |
| 20 | regular-file CLIs. No archive or compressed stream is unpacked. |
| 21 | |
| 22 | ## mzML and related mass-spectrometry formats |
| 23 | |
| 24 | HUPO-PSI identifies mzML 1.1.0 as the long-term stable format; its index schema |
| 25 | and controlled vocabulary continue to receive compatible updates. mzML is XML |
| 26 | with encoded binary arrays and controlled-vocabulary metadata. The current |
| 27 | official links use main schema 1.1.1 and optional indexed wrapper 1.1.3; distinguish |
| 28 | the stable specification from schema revision numbers. The IM-MS/DIA proposal |
| 29 | updated August 2026 has not yet entered the PSI Document Process. A generic XML |
| 30 | parser is not sufficient. Use a local validator: PSI reports the former online |
| 31 | validators no longer function. |
| 32 | |
| 33 | Use pinned pymzML, Pyteomics, OpenMS, or ProteoWizard tooling and inspect: |
| 34 | |
| 35 | - schema/version, controlled-vocabulary terms, source files, checksums, and |
| 36 | conversion software; |
| 37 | - run/instrument configuration, polarity, scan modes, MS levels, isolation, |
| 38 | activation, and data processing; |
| 39 | - spectrum/chromatogram counts, retention/mobility time, m/z and intensity |
| 40 | array lengths, precision, compression, and units; |
| 41 | - profile versus centroid data, TIC/BPC, calibration, lock mass, blanks, pooled |
| 42 | QC, standards, carryover, drift, and batch order; and |
| 43 | - truncated scans, empty arrays, non-finite values, and metadata consistency. |
| 44 | |
| 45 | Do not describe mzXML, mzData, mzMLb, or vendor RAW as equivalent to mzML. |
| 46 | Conversion can alter metadata, precision, centroiding, and compression; record |
| 47 | the converter/version/options and retain the original. |
| 48 | |
| 49 | ## JCAMP-DX |
| 50 | |
| 51 | IUPAC describes JCAMP-DX as a family of standards for spectral data exchange. |
| 52 | It has technique- and version-specific specifications (IR, NMR, MS, IMS, and |
| 53 | others). Choose the applicable published recommendation instead of assuming |
| 54 | all JCAMP files share one interchangeable schema. |
| 55 | |
| 56 | Before parsing, identify the technique and specification/version. Validate: |
| 57 | |
| 58 | - label/value records and required metadata; |
| 59 | - X/Y units, first/last X, point count, spacing, factors, and encoded numeric |
| 60 | representation; |
| 61 | - NTUPLES versus simpler XY forms; |
| 62 | - page/block boundaries and compound/instrument identifiers; and |
| 63 | - whether data are absorbance, transmittance, counts, complex NMR, peaks, or |
| 64 | continuous spectra. |
| 65 | |
| 66 | Metadata and comments are untrusted and should not be copied into a report. |
| 67 | The generic tabular scanner is not a JCAMP parser. |
| 68 | |
| 69 | ## NMR data |
| 70 | |
| 71 | `.fid`, Bruker directory layouts, Varian/Agilent layouts, processed spectra, |
| 72 | and NMR exchange files require producer-aware tooling such as nmrglue. Record: |
| 73 | |
| 74 | - vendor/software/version and complete acquisition directory; |
| 75 | - nucleus, field strength, spectral width, dwell time, point count, quadrature, |
| 76 | digital filter, scans, temperature, pulse sequence, and reference; |
| 77 | - raw FID versus processed spectrum, apodization, zero filling, Fourier |
| 78 | transform, phase, baseline, referencing, and solvent suppression; |
| 79 | - dimensional axes/units and whether data are real, imaginary, magnitude, or |
| 80 | complex; and |
| 81 | - sample preparation, concentration, pH, replicates, and batch/order. |
| 82 | |
| 83 | Peak picking, integration, baseline correction, phase correction, alignment, |
| 84 | binning, and normalization are transformations. Preserve raw data and report |
| 85 | parameter sensitivity; do not apply them automatically. |
| 86 | |
| 87 | ## Optical, vibrational, and diffraction spectra |
| 88 | |
| 89 | SPC, OPUS, WDF, SPE, instrument `.raw`, `.dat`, and text exports are |
| 90 | producer/variant dependent. Confirm: |
| 91 | |
| 92 | - physical X axis (wavelength, wavenumber, energy, angle, time) and units; |
| 93 | - Y quantity (counts, intensity, absorbance, transmittance, reflectance) and |
| 94 | calibration; |
| 95 | - point order/spacing, detector/channel, exposure/accumulations, resolution, |
| 96 | slit/grating/laser/source, and polarization; |
| 97 | - background/reference/dark correction and all processing already applied; and |
| 98 | - maps, time series, replicate spectra, and spatial coordinates. |
| 99 | |
| 100 | For XRD, crystallographic CIF/MTZ/HKL are also reference-only and need |
| 101 | crystallography-aware validation. A `.cif` suffix is ambiguous between |
| 102 | small-molecule CIF and PDBx/mmCIF. |
| 103 | |
| 104 | ## Chromatography and thermal/electrochemical exports |
| 105 | |
| 106 | Generic CSV/TSV can contain retention time, temperature, potential, wavelength, |
| 107 | or another independent axis. The general scripts can profile the table only |
| 108 | after the data dictionary confirms: |
| 109 | |
| 110 | - axis and signal columns, units, ordering, spacing, and replicate layout; |
| 111 | - blanks, calibration standards, internal standards, dilution factors, |
| 112 | injection order, batch, and sample identifiers; |
| 113 | - LOD/LOQ, saturation, censoring qualifiers, and negative/zero handling; and |
| 114 | - whether peaks/integrals are raw, manually edited, or software-derived. |
| 115 | |
| 116 | Do not infer an axis from monotonic values or a column name. Do not |
| 117 | automatically smooth, baseline-correct, align, integrate, normalize, subtract |
| 118 | blanks, or delete peaks. |
| 119 | |
| 120 | ## Safe bounded tabular workflow |
| 121 | |
| 122 | For an approved values-only export: |
| 123 | |
| 124 | ```bash |
| 125 | python scripts/tabular_profile.py spectrum.csv \ |
| 126 | --root /approved/project \ |
| 127 | --max-rows 100000 |
| 128 | |
| 129 | python scripts/missingness_leakage_audit.py spectrum.csv \ |
| 130 | --root /approved/project \ |
| 131 | --group-column sample_group \ |
| 132 | --entity-column sample_id \ |
| 133 | --split-column split \ |
| 134 | --time-column acquisition_time |
| 135 | |
| 136 | python scripts/distribution_sensitivity.py spectrum.csv \ |
| 137 | --root /approved/project \ |
| 138 | --column intensity |
| 139 | ``` |
| 140 | |
| 141 | Use only pseudonymous column roles in shared commands/logs. The outputs contain |
| 142 | aggregates and tokens, not spectra or identifiers. |
| 143 | |
| 144 | ## Analytical EDA rigor |
| 145 | |
| 146 | 1. Define the independent unit: scan, injection, spectrum, sample, batch, |
| 147 | subject, instrument, site, or experiment. |
| 148 | 2. Preserve raw acquisition files and processing audit trails. |
| 149 | 3. Record calibration, units, standards, blanks, internal standards, |
| 150 | acquisition order, maintenance, software, and method versions. |
| 151 | 4. Keep non-detects, below-LOQ values, saturation, missing scans, failed QC, and |
| 152 | true zeros distinct. Preserve qualifier and limit fields. |
| 153 | 5. Compare raw and processed summaries and sensitivity to baseline, smoothing, |
| 154 | peak picking, alignment, integration, normalization, and transformations. |
| 155 | 6. Investigate outliers against calibration, instrument state, carryover, and |
| 156 | sample handling; do not delete automatically. |
| 157 | 7. Split independent samples/batches/time before learned preprocessing. Never |
| 158 | fit normalization or feature selection on test data. |
| 159 | 8. Account for repeated spectra, technical replicates, correlated wavelengths/ |
| 160 | peaks, and many comparisons. |
| 161 | 9. Label discovered peaks/patterns as exploratory and confirm independently. |
| 162 | 10. Do not make identity, purity, mechanism, exposure, diagnostic, or causal |
| 163 | claims from EDA alone. |
| 164 | |
| 165 | ## Detection limits and censoring |
| 166 | |
| 167 | EPA guidance treats non-detects/over-detects as censored observations carrying |
| 168 | partial information and recommends preserving detection condition and limit |
| 169 | type rather than forcing a numeric result. Apply the same principle to |
| 170 | instrumental assays: |
| 171 | |
| 172 | - keep measured value, qualifier, limit type, and limit value in distinct |
| 173 | fields; |
| 174 | - do not replace censored values automatically with zero, LOD/2, or LOQ; |
| 175 | - summarize the censoring fraction by group/batch/time; |
| 176 | - choose a model appropriate to censoring and scientific design; and |
| 177 | - report sensitivity to plausible assumptions. |
| 178 | |
| 179 | ## Authoritative sources |
| 180 | |
| 181 | Current parser APIs and format guidance reviewed 2026-09-30. Historical |
| 182 | methodology citations are retained as source context. |
| 183 | |
| 184 | - HUPO-PSI, [mzML specification/status](https://www.psidev.info/mzml) |
| 185 | (mzML 1.1.0 long-term stable; current schema/CV links and 2026 IM-MS/DIA |
| 186 | proposal status). |
| 187 | - HUPO-PSI, [mzML GitHub specification repository](https://github.com/HUPO-PSI/mzML). |
| 188 | - IUPAC, [JCAMP-DX digital standard family](https://iupac.org/what-we-do/digital-standards/jcamp-dx/) |
| 189 | (page dated 2021-08-03; finalized technique-specific standards). |
| 190 | - IUPAC, [JCAMP-DX 5.01 recommendation](https://doi.org/10.1351/pac199971081549). |
| 191 | - nmrglue, [current documentation](https://nmrglue.readthedocs.io/en/latest/). |
| 192 | - US EPA, [Detection Limits Best Practices Guide](https://www.epa.gov/system/files/documents/2025-09/wqxdetectionlimitsbestpracticesguide_final.pdf), |
| 193 | dated August 2025. |
| 194 | - NIST/SEMATECH, [Exploratory Data Analysis](https://www.itl.nist.gov/div898/handbook/eda/eda.htm). |
| 195 | - FDA/ICH E9(R1), [estimands and sensitivity analysis](https://www.fda.gov/regulatory-information/search-fda-guidance-documents/e9r1-statistical-principles-clinical-trials-addendum-estimands-and-sensitivity-analysis-clinical), |
| 196 | final guidance May 2021. |
文件内容是平台检查时读出的文本，只供查看。平台不会运行包里的任何脚本或程序。
