references/proteomics_metabolomics_formats.md
9.6 KB查看排版效果| 1 | # Proteomics and Metabolomics Formats |
| 2 | |
| 3 | **Reviewed:** 2026-09-30 |
| 4 | **Executable scope:** No omics-native standard is parsed by bundled scripts. |
| 5 | Rectangular CSV/TSV result exports can use the general tabular CLIs after the |
| 6 | schema, units, and missing/censoring codes are confirmed. |
| 7 | |
| 8 | ## Exact capability boundary |
| 9 | |
| 10 | | Format | Bundled native inspection | Status | |
| 11 | |---|---|---| |
| 12 | | mzML/mzXML, vendor RAW | No | Reference-only MS tooling; see `spectroscopy_analytical_formats.md` | |
| 13 | | mzIdentML (`.mzid`, `.mzIdentML`) | No | Reference-only PSI schema/CV-aware tooling | |
| 14 | | mzTab 1.0 / mzTab-M 2.0 | No | Reference-only version-aware validator; generic TSV parsing is insufficient | |
| 15 | | pepXML/protXML | No | Reference-only search/inference-aware parser | |
| 16 | | featureXML/consensusXML/idXML | No | Reference-only OpenMS tooling | |
| 17 | | Rectangular `.csv`/`.tsv` feature or abundance table | General scripts | Bounded aggregate tabular EDA, no omics semantics | |
| 18 | | `.h5`/`.hdf5` | Generic metadata only | No payload values or convention validation | |
| 19 | | `.h5ad`, `.loom` | No semantic support | See bioinformatics reference | |
| 20 | | Pickled models/results | **Never** | Request non-executable export | |
| 21 | |
| 22 | Unknown formats fail closed. No format is identified from free-text metadata or |
| 23 | content guessing. |
| 24 | |
| 25 | ## mzML and raw spectra |
| 26 | |
| 27 | mzML is a HUPO-PSI standard for spectra/chromatograms; use PSI-aware tooling. |
| 28 | Vendor RAW extensions are ambiguous and often require vendor libraries or |
| 29 | conversion. Preserve originals and record converter, version, options, and |
| 30 | checksums. |
| 31 | |
| 32 | For spectral EDA, inventory: |
| 33 | |
| 34 | - acquisition method, instrument, polarity, MS levels, scan modes, precursor |
| 35 | isolation/activation, resolution, and centroid/profile status; |
| 36 | - run order, batches, blanks, pooled QC, standards, carryover, drift, and |
| 37 | calibration; |
| 38 | - spectrum/chromatogram counts, retention/mobility ranges, m/z coverage, TIC/ |
| 39 | BPC, peak counts, and missing/corrupt scans; and |
| 40 | - processing history, controlled-vocabulary terms, source files, and units. |
| 41 | |
| 42 | Do not automatically centroid, denoise, recalibrate, align, peak-pick, or |
| 43 | discard spectra. |
| 44 | |
| 45 | ## Identification formats |
| 46 | |
| 47 | ### mzIdentML |
| 48 | |
| 49 | mzIdentML represents peptide/protein identification results, scores, search |
| 50 | parameters, databases, modifications, and links to spectra using controlled |
| 51 | vocabularies. Validate the schema and CV mapping with PSI-aware tooling. |
| 52 | |
| 53 | Check: |
| 54 | |
| 55 | - search engine/version, sequence database/version, decoy strategy, enzyme, |
| 56 | tolerances, fixed/variable modifications, and spectrum references; |
| 57 | - score direction/meaning, rank, charge, mass error, peptide-spectrum matches, |
| 58 | peptides, proteins, and protein groups; |
| 59 | - target/decoy and FDR method at each reported level; and |
| 60 | - ambiguity from shared peptides, indistinguishable proteins, and inference. |
| 61 | |
| 62 | A score threshold is not automatically a validated FDR threshold. Do not |
| 63 | recompute or reinterpret confidence without the method and decoy design. |
| 64 | |
| 65 | ### pepXML/protXML |
| 66 | |
| 67 | These formats are Trans-Proteomic Pipeline conventions. Use Pyteomics or TPP |
| 68 | tools with the generating software/version known. Preserve search-engine, |
| 69 | PeptideProphet/ProteinProphet, modification, decoy, and inference context. |
| 70 | |
| 71 | ## mzTab and mzTab-M |
| 72 | |
| 73 | HUPO-PSI lists: |
| 74 | |
| 75 | - mzTab 1.0.0 as the final proteomics release (accepted June 2014); and |
| 76 | - mzTab-M 2.0.0 as the final metabolomics/small-molecule release (accepted |
| 77 | March 2019). |
| 78 | |
| 79 | mzTab-M 2.1.0 is listed as draft, not a final standard. Do not silently treat |
| 80 | it as 2.0. The draft has moved to the separate |
| 81 | [HUPO-PSI/mzTab-M repository](https://github.com/HUPO-PSI/mzTab-M); the released |
| 82 | 2.0 materials remain in HUPO-PSI/mzTab. |
| 83 | |
| 84 | Although mzTab is tab-delimited, it has section-specific row types, metadata, |
| 85 | controlled vocabulary, optional columns, and null conventions. The generic |
| 86 | rectangular TSV scanner is not a validator and will reject legitimate |
| 87 | non-rectangular section structure. Use the PSI specification/reference |
| 88 | validator, then export a controlled rectangular analysis table if needed. |
| 89 | |
| 90 | ## Rectangular quantitative tables |
| 91 | |
| 92 | Common outputs contain features/peptides/proteins/metabolites in rows and |
| 93 | samples in columns, or long-form measurements. Before using general CLIs, |
| 94 | create a data dictionary that records: |
| 95 | |
| 96 | - row entity and identifier namespace/version; |
| 97 | - sample/subject/specimen, condition, batch, injection order, and QC role; |
| 98 | - abundance scale (raw intensity, area, count, ratio, normalized/logged); |
| 99 | - zero, missing, censored, filtered, not-identified, and not-quantified codes; |
| 100 | - normalization, transformation, imputation, roll-up, and batch correction |
| 101 | already applied; |
| 102 | - internal standards, dilution, LOD/LOQ, blank subtraction, and detection |
| 103 | frequency; and |
| 104 | - peptide-to-protein or feature-to-metabolite ambiguity. |
| 105 | |
| 106 | Do not assume zeros are measured zeros. Missingness is often abundance-, |
| 107 | feature-, batch-, or identification-dependent and may be non-random. |
| 108 | |
| 109 | ### Safe commands |
| 110 | |
| 111 | ```bash |
| 112 | python scripts/tabular_profile.py abundance.csv \ |
| 113 | --root /approved/project \ |
| 114 | --missing-token NA \ |
| 115 | --max-rows 100000 |
| 116 | |
| 117 | python scripts/missingness_leakage_audit.py abundance.csv \ |
| 118 | --root /approved/project \ |
| 119 | --group-column condition \ |
| 120 | --entity-column subject_id \ |
| 121 | --split-column split \ |
| 122 | --time-column acquisition_time |
| 123 | |
| 124 | python scripts/distribution_sensitivity.py abundance.csv \ |
| 125 | --root /approved/project \ |
| 126 | --column intensity |
| 127 | ``` |
| 128 | |
| 129 | The column arguments are exact local identifiers; output tokenizes them unless |
| 130 | `--reveal-identifiers` is explicit. Values and subject/sample identifiers are |
| 131 | not emitted. |
| 132 | |
| 133 | ## Missingness, censoring, and limits |
| 134 | |
| 135 | Separate at least: |
| 136 | |
| 137 | - structurally absent/not applicable; |
| 138 | - not detected; |
| 139 | - detected below quantitation; |
| 140 | - failed identification or confidence filter; |
| 141 | - failed extraction/integration; |
| 142 | - filtered during preprocessing; |
| 143 | - saturated/above range; and |
| 144 | - genuinely missing metadata. |
| 145 | |
| 146 | Preserve flags and limits in separate columns. Do not automatically replace |
| 147 | non-detects with zero, half-minimum, LOD/2, or a random draw. Report missing/ |
| 148 | censored fractions by feature, sample, condition, batch, and run order, and |
| 149 | compare conclusions across scientifically justified handling strategies. |
| 150 | |
| 151 | ## Distribution and outlier sensitivity |
| 152 | |
| 153 | For abundance tables: |
| 154 | |
| 155 | - inspect sample totals/detection rates and feature detection frequency; |
| 156 | - compare raw-scale and scientifically justified log/variance-stabilizing |
| 157 | diagnostics without overwriting raw data; |
| 158 | - compare mean/SD with median/IQR/MAD and leave-one-sample/batch sensitivity; |
| 159 | - investigate outliers against blank/QC/internal-standard performance, |
| 160 | acquisition order, contamination, carryover, and sample handling; and |
| 161 | - preserve excluded samples/features with reasons and show sensitivity. |
| 162 | |
| 163 | PCA/clustering can reveal structure but is not proof of batch, identity, or |
| 164 | biological separation. Fit transformations and feature selection on training |
| 165 | data only. |
| 166 | |
| 167 | ## Design, leakage, and inference |
| 168 | |
| 169 | 1. Define the independent experimental unit; technical injections, spectra, |
| 170 | peptides, or features are usually not independent subjects. |
| 171 | 2. Preserve subject/sample pairing, repeated measures, batches, sites, and |
| 172 | acquisition order. |
| 173 | 3. Split by subject/specimen/batch/time before normalization, imputation, |
| 174 | feature selection, PCA, or model tuning. |
| 175 | 4. Ensure spectra/peptides/features derived from one sample do not cross |
| 176 | train/test boundaries. |
| 177 | 5. Distinguish QC, blank, pooled, calibrator, and biological samples. |
| 178 | 6. Treat identification/feature discovery and differential testing as separate |
| 179 | selection stages when assessing error rates. |
| 180 | 7. Define the hypothesis family (features, contrasts, endpoints) and report |
| 181 | effect sizes/uncertainty plus an appropriate FWER/FDR method. |
| 182 | 8. Label discoveries from EDA as exploratory and confirm on independent data. |
| 183 | 9. Do not make biomarker, diagnostic, mechanism, exposure, or causal claims |
| 184 | from descriptive patterns. |
| 185 | |
| 186 | ## HDF5 and related containers |
| 187 | |
| 188 | The generic HDF5 inspector reports only bounded hierarchy/dataset metadata. It |
| 189 | does not: |
| 190 | |
| 191 | - read spectra, abundance matrices, annotations, or attributes; |
| 192 | - follow soft/external links or external dataset storage; |
| 193 | - validate mzMLb, H5AD, Loom, or vendor schemas; or |
| 194 | - invoke filter plugins for dataset decompression. |
| 195 | |
| 196 | Use the convention's official reader/validator for semantics. NumPy object |
| 197 | arrays and all pickle-based objects are rejected. |
| 198 | |
| 199 | ## Authoritative sources |
| 200 | |
| 201 | Current parser APIs and format guidance reviewed 2026-09-30. Historical |
| 202 | methodology citations are retained as source context. |
| 203 | |
| 204 | - HUPO-PSI, [mzML specification/status](https://www.psidev.info/mzml) |
| 205 | (mzML 1.1.0 long-term stable). |
| 206 | - HUPO-PSI, [mzIdentML](https://www.psidev.info/mzidentml). |
| 207 | - HUPO-PSI, [mzTab specifications](https://www.psidev.info/mztab-specifications) |
| 208 | (page updated 2024-04-19; mzTab 1.0.0 final, mzTab-M 2.0.0 final, |
| 209 | mzTab-M 2.1.0 draft). |
| 210 | - HUPO-PSI, [mzTab repository and released specifications](https://github.com/HUPO-PSI/mzTab). |
| 211 | - Hoffmann et al. (2019), [mzTab-M 2.0](https://doi.org/10.1021/acs.analchem.8b04310), |
| 212 | published 2019-01-28. |
| 213 | - Pyteomics, [formats documentation](https://pyteomics.readthedocs.io/en/latest/). |
| 214 | - OpenMS, [recognized file types](https://openms.de/documentation/structOpenMS_1_1FileTypes.html). |
| 215 | - US EPA, [Detection Limits Best Practices Guide](https://www.epa.gov/system/files/documents/2025-09/wqxdetectionlimitsbestpracticesguide_final.pdf), |
| 216 | dated August 2025. |
| 217 | - FDA/ICH E9(R1), [sensitivity analysis guidance](https://www.fda.gov/regulatory-information/search-fda-guidance-documents/e9r1-statistical-principles-clinical-trials-addendum-estimands-and-sensitivity-analysis-clinical), |
| 218 | final May 2021. |
| 219 | - Benjamini and Hochberg (1995), [FDR control](https://academic.oup.com/jrsssb/article/57/1/289/7035855). |
| 220 | - scikit-learn, [data leakage guidance](https://scikit-learn.org/stable/common_pitfalls.html). |
文件内容是平台检查时读出的文本，只供查看。平台不会运行包里的任何脚本或程序。
