探索性数据分析

Exploratory Data Analysis

在本地检查 CSV、TSV、JSON 等数据的字段、缺失值、分布、异常值和数据质量。

灵栈官方最新版本 0.1.0更新于 2026-10-16下载 2 次

CSV
数据质量
缺失值
异常值
数据审计
科研数据

references/general_scientific_formats.md

1	# General Scientific Formats and EDA Rigor
2
3	Reviewed: 2026-09-30
4	Scope: Exact capabilities of the bundled scripts plus conservative,
5	documented workflows for common tabular and array containers.
6
7	## Capability boundary
8
9	\| Format \| Bundled executable inspection \| Depth \|
10	\|---\|---\|---\|
11	\| `.csv`, `.tsv` \| Yes, Python standard library \| Bounded UTF-8 rectangular scan; schema, missingness, aggregate statistics, duplicate hashes, group/split leakage, and sensitivity \|
12	\| `.json` \| Yes, Python standard library \| Bounded strict whole-document parse; structure and type counts only \|
13	\| `.npy` \| Optional, `numpy==2.5.3` \| Header/shape/dtype plus bounded numeric sample; `allow_pickle=False` \|
14	\| `.npz` \| Optional, `numpy==2.5.3` \| ZIP member/size/ratio preflight, then bounded per-array inspection; `allow_pickle=False` \|
15	\| `.h5`, `.hdf5` \| Optional, `h5py==3.16.0` \| Bounded hierarchy and dataset metadata; payloads, attributes, soft links, external links, and external storage are not read \|
16	\| `.parquet`, `.feather` \| No \| Reference-only pandas/Polars/Arrow workflow \|
17	\| `.xlsx`, `.xls` \| No \| Reference-only workbook review; formulas, links, hidden content, and macros require separate handling \|
18	\| `.zarr`, `.nc`, `.mat`, `.fits` \| No \| Reference-only domain tooling \|
19	\| Pickle/joblib/dill \| Never \| Deserialization is outside this skill's security boundary \|
20
21	“Bundled executable” means a bounded inspection exists; it does not mean
22	complete-file semantic validation. Unknown suffixes fail closed. Compressed
23	generic archives are not unpacked.
24
25	## Safe local-file contract
26
27	All bundled CLIs:
28
29	1. accept only regular local files inside an explicit `--root`;
30	2. reject URLs, `..` traversal, home expansion, symlinks, multiply linked
31	inputs, and special files;
32	3. enforce byte, row, field, column, member, object, and report limits;
33	4. use the registered suffix and, where unambiguous, verify a magic signature;
34	5. never use generic binary/text guessing as a fallback;
35	6. emit aggregate statistics and tokenized identifiers by default, never rows;
36	7. treat labels, headers, metadata, and file text as untrusted data, not
37	instructions; and
38	8. write private (`0600`) outputs atomically and refuse overwrite unless
39	`--force` is explicit.
40
41	Hashes/tokens are deterministic pseudonyms, not anonymization. A file hash or a
42	low-cardinality value token can still be linkable.
43
44	## CSV and TSV
45
46	### Bundled approach
47
48	`tabular_profile.py`, `missingness_leakage_audit.py`, and
49	`distribution_sensitivity.py` use Python's `csv` module with:
50
51	- UTF-8/UTF-8-with-BOM decoding and strict errors;
52	- a fixed delimiter selected from `.csv` or `.tsv`, not sniffed;
53	- `strict=True`, a bounded `csv.field_size_limit`, fixed maximum columns, and
54	rectangular-row enforcement;
55	- an explicit missing-code policy (empty/whitespace only unless the user adds
56	`--missing-token`);
57	- streaming Welford moments and deterministic bounded samples; and
58	- no row or raw categorical-value output.
59
60	Delimiter, decimal convention, thousands separators, encodings, comment
61	syntax, and missing codes are part of the data dictionary. Do not silently
62	guess them.
63
64	### pandas 3.0.6 (documented alternate backend)
65
66	PyPI published `pandas==3.0.6` on 2026-09-17; it is the tested alternate backend.
67	When pandas is appropriate, preserve the same outer path/size checks and use
68	bounded selections:
69
70	```python
71	import pandas as pd
72
73	frame = pd.read_csv(
74	local_path,
75	nrows=100_000,
76	usecols=approved_columns,
77	dtype=declared_types,
78	na_values=declared_missing_codes,
79	keep_default_na=False,
80	on_bad_lines="error",
81	)
82	```
83
84	`nrows` and `usecols` reduce work, but do not replace file-size, field-size, or
85	privacy controls. Keep parsing errors visible. Do not use `on_bad_lines="skip"`
86	for EDA because it changes the analyzed population.
87
88	### Polars 1.44.2 (documented alternate backend)
89
90	PyPI published `polars==1.44.2` on 2026-09-09. Current `polars.read_csv`
91	supports `columns`, `schema`, `schema_overrides`, `null_values`,
92	`infer_schema_length`, and `n_rows`. Its docs note that:
93
94	- malformed non-RFC-4180 data may have undefined behavior;
95	- `ignore_errors=False` is the safe default;
96	- `infer_schema_length=None` scans the full data into memory; and
97	- with multithreaded parsing, `n_rows` is not guaranteed as a strict upper
98	bound.
99
100	Prevalidate a local path; do not pass URLs or rely on optional `fsspec`. For
101	strict bounded EDA, the bundled standard-library scanner is the reference
102	implementation.
103
104	## Strict JSON
105
106	Python's current `json` documentation warns that malicious JSON can consume
107	substantial CPU and memory and recommends limiting input size. It also
108	documents that the default decoder accepts `NaN`/`Infinity` and silently keeps
109	the last duplicate object key.
110
111	The bundled inspector therefore:
112
113	- caps the file at 16 MiB for parsing;
114	- requires UTF-8;
115	- rejects duplicate keys, non-finite constants, and decimal/exponent literals
116	that overflow finite binary64 (for example `1e400`);
117	- catches recursion/resource errors;
118	- traverses at most 100,000 nodes; and
119	- emits only root type, depth, type counts, collection sizes, and tokenized
120	top-level field identifiers.
121
122	The decimal range restriction is a tool limit, not a claim that every larger
123	number violates the JSON grammar. JSON Lines/NDJSON is not registered.
124	Rename-and-guess is not allowed.
125
126	## NumPy NPY and NPZ
127
128	NumPy's NPY specification stores shape and dtype in a header. NPZ is a ZIP
129	archive whose members are NPY files. Object arrays can contain pickled Python
130	objects.
131
132	The bundled inspector always uses:
133
134	```python
135	array = np.load(
136	local_path,
137	mmap_mode="r",
138	allow_pickle=False,
139	max_header_size=10_000,
140	)
141	```
142
143	For NPZ it first rejects:
144
145	- non-NPY members, directories, traversal paths, encryption, and duplicate or
146	excessive members;
147	- declared uncompressed content above 128 MiB; and
148	- a per-member compression ratio above 100.
149
150	It then loads one array at a time with `allow_pickle=False`. Numeric summaries
151	use at most 4,096 deterministic sample elements. Large arrays use indexed
152	`ndarray.flat` reads, avoiding a full C-order copy for Fortran-contiguous NPY
153	memory maps. Structured dtype field names
154	are identifiers and are tokenized by default. Object dtype is rejected; there
155	is no `allow_pickle` override.
156
157	Memory mapping reduces array payload reads but does not make malformed headers
158	or huge shapes harmless. The outer byte and header limits remain mandatory.
159
160	## HDF5 and h5py
161
162	HDF5 is a container, not a semantic schema. Generic HDF5 inspection does not
163	validate AnnData/H5AD, Loom, Imaris, mzMLb, or a laboratory's custom layout.
164
165	h5py documents hard, soft, and external links. Dereferencing an external link
166	opens another file. The bundled inspector uses `getlink=True` to classify
167	links and never follows soft or external links. It:
168
169	- reports at most 1,000 objects and 16 group levels;
170	- deduplicates hard-link aliases;
171	- reports shapes, dtype classes, chunking, compression presence, virtual/external
172	storage flags, and attribute counts;
173	- does not read dataset payloads or attribute values;
174	- does not call array conversion, user-defined callbacks, or dynamic
175	evaluation; and
176	- does not invoke HDF5 filter plugins to decode data.
177
178	Do not copy external-link filenames, object names, or attributes into reports.
179	Do not set or trust `HDF5_PLUGIN_PATH` for untrusted files.
180
181	## Reference-only formats
182
183	### Parquet and Feather
184
185	Use a pinned Arrow/pandas/Polars environment after local path validation.
186	Inspect schema and row-group metadata first, select approved columns, and bound
187	rows. The bundled scripts do not parse these formats, so they are not part of
188	automated support.
189
190	### Excel
191
192	Spreadsheets can contain formulas, external links, hidden sheets, names,
193	comments, and macros. Never enable macros, formula evaluation, or linked-data
194	refresh. Export a values-only review copy to CSV/TSV after a human validates
195	sheet choice, units, formulas, and merged/hidden regions. Preserve the original.
196
197	### Zarr and directory stores
198
199	Zarr/OME-Zarr are directory or object-store layouts rather than single regular
200	files. The local-file CLIs reject directories. Use a separately sandboxed,
201	version-aware Zarr workflow with explicit store and codec allowlists.
202
203	## Statistical EDA contract
204
205	1. Preserve the raw file and create a data dictionary with units and provenance.
206	2. Identify observational units, replicates, grouping, pairing, clustering,
207	batches, sites, and time order before pooling.
208	3. Preserve missingness and censoring indicators. Do not automatically impute,
209	substitute LOD/2, or treat non-detects as zero.
210	4. Compare classical and robust summaries. Outlier flags trigger measurement
211	review and sensitivity analysis, not automatic deletion.
212	5. Record transformation formulas and scientific rationale; fit any learned
213	parameter on training data only and retain raw-scale results.
214	6. Split subjects/groups/time before fitting imputers, scalers, feature
215	selection, PCA, or other preprocessing.
216	7. Label post hoc patterns as exploratory. Define the hypothesis family and
217	FWER/FDR plan before confirmatory testing.
218	8. Report effect sizes, uncertainty, assumptions, limitations, exact software
219	versions, commands, deterministic rules/seeds, and derived artifact hashes.
220	9. Do not make causal claims from descriptive associations.
221
222	## Pinned optional snapshot
223
224	Release metadata and synthetic runtime checks verified 2026-09-30:
225
226	```bash
227	uv pip install \
228	"numpy==2.5.3" \
229	"pandas==3.0.6" \
230	"polars==1.44.2" \
231	"h5py==3.16.0"
232	```
233
234	NumPy 2.5.3 requires Python 3.12+. These are direct-package snapshots, not a
235	transitive lock; record a lockfile for a real analysis.
236
237	## Authoritative sources
238
239	Current parser APIs and format guidance reviewed 2026-09-30. Historical
240	methodology citations are retained as source context.
241
242	- Python 3.14, [`csv` — CSV File Reading and Writing](https://docs.python.org/3/library/csv.html).
243	- Python 3.14, [`json` — JSON encoder and decoder](https://docs.python.org/3/library/json.html).
244	- NumPy 2.5, [input/output reference](https://numpy.org/doc/stable/reference/routines.io.html),
245	[`numpy.load`](https://numpy.org/doc/stable/reference/generated/numpy.load.html),
246	[NPY/NPZ format](https://numpy.org/doc/stable/reference/generated/numpy.lib.format.html),
247	[`ndarray.flat`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.flat.html),
248	and [security guidance](https://numpy.org/doc/stable/reference/security.html).
249	- pandas 3.0, [I/O tools](https://pandas.pydata.org/docs/user_guide/io.html);
250	[PyPI 3.0.6](https://pypi.org/project/pandas/), released 2026-09-17.
251	- Polars 1.44, [`polars.read_csv`](https://docs.pola.rs/api/python/stable/reference/api/polars.read_csv.html);
252	[PyPI 1.44.2](https://pypi.org/project/polars/), released 2026-09-09.
253	- h5py 3.16, [groups and links](https://docs.h5py.org/en/stable/high/group.html);
254	[PyPI 3.16.0](https://pypi.org/project/h5py/), released 2026-03-06.
255	- NIST/SEMATECH, [Exploratory Data Analysis](https://www.itl.nist.gov/div898/handbook/eda/eda.htm)
256	and [chapter references](https://www.itl.nist.gov/div898/handbook/eda/section4/eda43.htm).
257	- Box and Cox (1964), [“An Analysis of Transformations”](https://doi.org/10.1111/j.2517-6161.1964.tb00553.x).
258	- FDA/ICH E9(R1), [Estimands and Sensitivity Analysis](https://www.fda.gov/regulatory-information/search-fda-guidance-documents/e9r1-statistical-principles-clinical-trials-addendum-estimands-and-sensitivity-analysis-clinical),
259	final guidance May 2021.
260	- US EPA, [Detection Limits Best Practices Guide](https://www.epa.gov/system/files/documents/2025-09/wqxdetectionlimitsbestpracticesguide_final.pdf),
261	dated August 2025.
262	- scikit-learn, [common pitfalls and data leakage](https://scikit-learn.org/stable/common_pitfalls.html).
263	- Benjamini and Hochberg (1995), [false discovery rate](https://academic.oup.com/jrsssb/article/57/1/289/7035855).
264	- Wasserstein, Schirm, and Lazar (2019), [Moving to a World Beyond “p < 0.05”](https://doi.org/10.1080/00031305.2019.1583913).
265	- National Academies (2019), [Reproducibility and Replicability in Science](https://doi.org/10.17226/25303).
266	- Wilkinson et al. (2016), [FAIR Guiding Principles](https://doi.org/10.1038/sdata.2016.18).

文件内容是平台检查时读出的文本，只供查看。平台不会运行包里的任何脚本或程序。