DP44/enriched-qwen3-coder-next/Common/DTS.Common.Serialization/HDF.md

---
source_files:
  - Common/DTS.Common.Serialization/HDF/HDF.File.cs
  - Common/DTS.Common.Serialization/HDF/HDF.File.Writer.cs
generated_at: "2026-04-16T03:36:32.708668+00:00"
model: "Qwen/Qwen3-Coder-Next-FP8"
schema_version: 1
sha256: "6a62f69116272309"
---

# HDF

## Documentation: `DTS.Serialization.HDF.File`

---

### 1. Purpose

This module implements a concrete serialization backend for writing test data to HDF5 files, specifically tailored for DTS’s data acquisition systems. It extends `Serialization.File` and implements `IWritable<Test>`, enabling structured export of test metadata, raw sensor data (in ADC, mV, and Engineering Units), and associated binary files (logs, reports, setup, `.dts`) into a hierarchical HDF5 container. The implementation is tightly coupled to DTS’s internal data models (`Test`, `AnalogInputChannel`, `DataScaler`) and supports WIAMan-compliant metadata and S6 system attributes for interoperability with external analysis tools.

---

### 2. Public Interface

#### `public partial class File : Serialization.File, IWritable<Test>`

- **`File()`**  
  Constructor. Initializes the base class with `"HDF"` as the file type identifier.

- **`IWriter<Test> Exporter { get; }`**  
  Lazily instantiates and returns a `Writer` instance, using the default encoding. Throws an `Exception` wrapped around any inner exception encountered during writer creation.

- **`bool ExportADC { set; }`**  
  Controls whether raw ADC data is exported. Delegates to `((Writer)Exporter).ExportADC`.

- **`bool ExportEU { set; }`**  
  Controls whether engineering unit (EU) data is exported. Delegates to `((Writer)Exporter).ExportEU`.

- **`bool ExportMV { set; }`**  
  Controls whether millivolt (mV) data is exported. Delegates to `((Writer)Exporter).ExportMV`.

- **`bool ExportLogs { set; }`**  
  Controls whether log files (from `/Logs` directory) are included in the HDF5 `/Files/Logs` group. Delegates to `((Writer)Exporter).ExportLogs`.

- **`bool ExportReports { set; }`**  
  Controls whether report files (from `/Reports` directory) are included in `/Files/Reports`. Delegates to `((Writer)Exporter).ExportReports`.

- **`bool ExportSetup { set; }`**  
  Controls whether setup files (from `/SETUP` directory) are included in `/Files/SETUP`. Delegates to `((Writer)Exporter).ExportSetup`.

- **`bool ExportDTSFile { set; }`**  
  Controls whether `.dts` files (from `/Binary/{dataDir}`) are included in `/Files/Binary/{dataDir}`. Delegates to `((Writer)Exporter).ExportDTSFile`.

- **`string CustomerName { set; }`**  
  Sets WIAMan metadata attribute `Director`. Delegates to `((Writer)Exporter).CustomerName`.

- **`string TestEngineerName { set; }`**  
  Sets WIAMan metadata attribute `Operator`. Delegates to `((Writer)Exporter).TestEngineerName`.

- **`string LabName { set; }`**  
  Sets WIAMan metadata attribute `Location`. Delegates to `((Writer)Exporter).LabName`.

- **`bool IsWiamanData { set; }`**  
  Sets WIAMan metadata attribute `Is WIAMan Data`. Delegates to `((Writer)Exporter).IsWiamanData`.

- **`Dictionary<string, string> ISOToFineLocation3 { set; }`**  
  Used to populate WIAMan attribute `Anthropomorphic Label`. Delegates to `((Writer)Exporter).ISOToFineLocation3`.

- **`Dictionary<string, string> ISOToPhysicalDimension { set; }`**  
  Used to populate WIAMan attribute `Channel Label:Category`. Delegates to `((Writer)Exporter).ISOToPhysicalDimension`.

- **`Dictionary<string, string> ISOToPosition { set; }`**  
  Used to populate WIAMan attribute `Channel Label:Optional`. Delegates to `((Writer)Exporter).ISOToPosition`.

- **`Dictionary<string, string> ISOToTransducerMainLocation { set; }`**  
  *Not used in the current implementation.* Delegates to `((Writer)Exporter).ISOToTransducerMainLocation`, but the setter is present in `File` while the corresponding property is not declared in `Writer` (see *Gotchas*).

---

#### `public class Writer : Writer<File>, IWriter<Test>`

- **`internal File WriterParent { get; }`**  
  Reference to the owning `File` instance.

- **`bool ExportADC { get; set; }`**  
  Controls ADC data export. Default: `true`.

- **`bool ExportEU { get; set; }`**  
  Controls EU data export. Default: `true`.

- **`bool ExportMV { get; set; }`**  
  Controls mV data export. Default: `true`.

- **`bool ExportLogs { get; set; }`**  
  Controls log file inclusion. Default: `true`.

- **`bool ExportReports { get; set; }`**  
  Controls report file inclusion. Default: `true`.

- **`bool ExportSetup { get; set; }`**  
  Controls setup file inclusion. Default: `true`.

- **`bool ExportDTSFile { get; set; }`**  
  Controls `.dts` file inclusion. Default: `true`.

- **`string CustomerName { get; set; }`**  
  WIAMan metadata: `Director`.

- **`string TestEngineerName { get; set; }`**  
  WIAMan metadata: `Operator`.

- **`string LabName { get; set; }`**  
  WIAMan metadata: `Location`.

- **`bool IsWiamanData { get; set; }`**  
  WIAMan metadata: `Is WIAMan Data`.

- **`Dictionary<string, string> ISOToFineLocation3 { get; set; }`**  
  Used to construct `Anthropomorphic Label`.

- **`Dictionary<string, string> ISOToPhysicalDimension { get; set; }`**  
  Used to construct `Channel Label:Category`.

- **`Dictionary<string, string> ISOToPosition { get; set; }`**  
  Used to construct `Channel Label:Optional`.

- **`Dictionary<string, string> ISOToTransducerMainLocation { get; set; }`**  
  *Declared in `Writer` but never used.* (See *Gotchas*.)

- **`void Write(string pathname, string id, string dataFolder, Test test, bool bFiltering, bool includeGroupNameInISOExport, FilteredData fd, Test.Module.Channel tmChannel, int channelNumber, BeginEventHandler beginEventHandler, CancelEventHandler cancelEventHandler, EndEventHandler endEventHandler, TickEventHandler tickEventHandler, ErrorEventHandler errorEventHandler, CancelRequested cancelRequested, double minStartTime, int dataCollectionLength)`**  
  Main export method. Writes the `Test` object to an HDF5 file at `pathname`.  
  - Creates `/Files/{folder}` groups for logs, setup, reports, and `.dts` files if `Export*` flags are enabled.  
  - Writes three dataset groups: `/Datasets_EU`, `/Datasets`, `/Datasets_MV`, depending on flags.  
  - For each `AnalogInputChannel`, writes channel-specific metadata (WIAMan + S6 attributes) and raw data as datasets named `{index}: Strain_RawYData`.  
  - Uses `Parallel.ForEach` over `test.Channels` for performance.  
  - Progress is reported via `tickEventHandler` in 100-step increments.  
  - Errors are logged via `APILogger` and propagated via `errorEventHandler`.

- **`private Common.DAS.Concepts.DataScaler GetDataScaler(Test.Module.AnalogInputChannel currentAnalogChannel)`**  
  Constructs a `DataScaler` from channel properties (e.g., `UnitConversion`, `Multiplier`, `Sensitivity`, `ZeroMethod`, `RemovedADC`, `MeasuredExcitationVoltage`, etc.). Includes robust error handling with logging for individual scaler property assignments.

- **`private void CheckStatus(long status, ErrorLocation location)`**  
  Throws an `Exception` with `location.ToString()` if `status < 0`. Used after HDF5 API calls.

- **`private void CreateStringAttribute(long objectId, string attributeName, string attributeValue)`**  
  Creates an HDF5 string attribute on `objectId`. Handles null/empty values gracefully. Uses `H5S.create`, `H5T.copy`, `H5A.create`, `H5A.write`, and `Marshal.StringToHGlobalAnsi`. All handles are closed in `finally`.

- **`private void CreateIntAttribute(long objectId, string title, int value)`**  
  If `ATTRIBUTE_STRING_DATATYPE_ONLY` is `true` (hardcoded), converts `value` to string and delegates to `CreateStringAttribute`. Otherwise, would create numeric attribute (not implemented).

- **`private void CreateUlongAttribute(long objectId, string title, ulong value)`**  
  Same behavior as `CreateIntAttribute`.

- **`private void CreateDoubleAttribute(long objectId, string title, double value)`**  
  Same behavior as `CreateIntAttribute`.

- **`private void AddDirectoryIfExists(string binaryPath, string folder, long hdfObjectId, string fileExtension)`**  
  Adds all files matching `fileExtension` from a computed source directory (e.g., `root/Logs`) into an HDF5 group at `/Files/{folder}`. Uses nested `DirectoryInfo.Parent` checks to determine root. Reads files into byte arrays and writes them as 1D `NATIVE_UCHAR` datasets.

- **`private int ComputeNumberOfSteps(Test test)`**  
  Computes total export steps based on enabled export flags and channel count. Each enabled export type contributes `Channels.Count` steps; each file group (logs, setup, reports, dts) contributes 1 step.

- **`internal Writer(File fileType, int encoding)`**  
  Constructor. Initializes all `Export*` flags to `true`. Sets `WriterParent`.

- **`public void Initialize(...)`**  
  *Empty implementation.* No initialization logic is performed.

---

### 3. Invariants

- **HDF5 file structure**:  
  - `/Files/{folder}` groups are created only if corresponding `Export*` flag is `true`.  
  - `/Datasets`, `/Datasets_EU`, `/Datasets_MV` groups are created only if `ExportADC`, `ExportEU`, or `ExportMV` is `true`, respectively.  
  - Each channel’s data resides under a group named `/{datasetGroup}/{index:0000}: {channelName}`.  
  - Channel data datasets are named `{index}: Strain_RawYData`.

- **Data type in datasets**:  
  - ADC: `H5T.NATIVE_SHORT` (16-bit signed integer).  
  - mV/EU: `H5T.NATIVE_DOUBLE` (64-bit float).

- **Channel ordering**:  
  Channels are sorted by `AbsoluteDisplayOrder` before processing.

- **Progress reporting**:  
  Progress updates occur every `UPDATE_INTERVAL = 1000` samples per channel, and at group-level milestones (e.g., after each file group or dataset group is written). Final progress is always set to `100D` in `finally`.

- **WIAMan attribute requirement**:  
  All WIAMan attributes listed in code comments are written *only* if `IsWiamanData` is `true`?  
  → **No**: Attributes are written unconditionally. `IsWiamanData` only sets a string attribute `"Is WIAMan Data"` to `"TRUE"`/`"FALSE"`.

- **String attributes**:  
  All attributes (int, double, ulong) are stored as strings if `ATTRIBUTE_STRING_DATATYPE_ONLY = true` (hardcoded). No numeric HDF5 attributes are created.

- **Locking**:  
  HDF5 API calls are guarded by `H5WriteLock` (a private `object`). Progress updates use `_updateProgressLock`.

---

### 4. Dependencies

#### Imports/Usings:
- `HDF.PInvoke` — P/Invoke wrapper for HDF5 C library.
- `System.Runtime.InteropServices` — For `Marshal.AllocHGlobal`, `FreeHGlobal`, `StringToHGlobalAnsi`.
- `DTS.Common.Enums.Sensors` — For `SensorConstants.BridgeType`.
- `DTS.Common.Utilities.Logging` — For `APILogger`.
- `System.Collections.Generic`, `System.IO`, `System.Linq`, `System.Threading.Tasks` — Standard .NET.

#### Internal Dependencies:
- `Serialization.File` — Base class (inherited).
- `IWriter<Test>` — Interface implemented.
- `Test` — Core data model (from `DTS.Serialization` or similar).
- `Test.Module`, `Test.Module.AnalogInputChannel`, `FilteredData`, `DataScaler` — Internal DTS types.
- `BeginEventHandler`, `CancelEventHandler`, `EndEventHandler`, `TickEventHandler`, `ErrorEventHandler`, `CancelRequested` — Delegates for progress/cancellation.

#### External Dependencies:
- HDF5 library (via `HDF.PInvoke`).
- File system access for reading logs, setup, reports, and `.dts` files.

---

### 5. Gotchas

- **`ISOToTransducerMainLocation` is unused**:  
  The `File` class exposes a setter for `ISOToTransducerMainLocation`, and the `Writer` class declares a corresponding property, but the property is never referenced in the code. This is likely dead code.

- **`Initialize` method is empty**:  
  The `Writer.Initialize(...)` method has no implementation. All work is done in `Write(...)`. This may violate expectations of a two-phase initialization pattern.

- **String-only attributes**:  
  All numeric attributes (int, double, ulong) are stored as strings due to `ATTRIBUTE_STRING_DATATYPE_ONLY = true`. This increases file size and complicates downstream parsing.

- **Nested `DirectoryInfo.Parent` checks**:  
  The `AddDirectoryIfExists` method uses a brittle chain of `Parent.Parent.Parent` to determine the root directory. This assumes a fixed directory structure and may fail if the path depth is insufficient.

- **Parallel channel processing**:  
  Channels are processed in parallel (`Parallel.ForEach`), but progress updates are *not* thread-safe beyond the `_updateProgressLock`. Steps completed (`stepsCompleted`) is incremented outside the lock, risking race conditions and incorrect progress reporting.

- **Hardcoded dataset names**:  
  All channel datasets are named `{index}: Strain_RawYData`, regardless of data type (ADC/mV/EU). This may cause confusion or overwriting if multiple dataset groups exist.

- **No cancellation support in `Write`**:  
  Although `cancelRequested` is passed to `Write`, it is never checked. The export runs to completion regardless of cancellation.

- **Error handling is lossy**:  
  `CheckStatus` throws a generic `Exception` with only the location name (e.g., `"File"`), losing the underlying HDF5 error code or message. Detailed diagnostics rely on `APILogger.Log` calls.

- **Memory allocation per sample**:  
  Data is written sample-by-sample using `Marshal.WriteByte` in loops. This is inefficient compared to bulk copy (e.g., `Marshal.Copy`). The comment references performance fixes, but the implementation remains suboptimal.

- **No validation of `ISOTo*` dictionaries**:  
  Accessing `ISOToFineLocation3[aic.IsoCode]` without checking for key existence will throw `KeyNotFoundException` if the dictionary is incomplete or keys are missing.

- **`minStartTime` and `dataCollectionLength` are unused**:  
  These parameters appear in `Write` signatures but are never used in the implementation.

- **`ExtensionPrefix` is settable but unused**:  
  The `ExtensionPrefix` property is defined but not used in filename construction (`{id}_{LabName}_1of1{(ExtensionPrefix ?? "")}.h5`), so it only affects the filename if explicitly set to a non-empty value.

- **No support for digital or calculated channels**:  
  `Parallel.ForEach` skips non-`AnalogInputChannel` instances (`if (null == aic) return;`). Digital channels, calculated channels, or other channel types are silently ignored.

- **Time handling assumptions**:  
  `Reference Time` is computed from `TriggerTimestampSec` and `TriggerTimestampNanoSec`, assuming Unix epoch. No timezone or leap-second handling is evident.

- **Hardcoded export version**:  
  `HDF_EXPORT_VERSION = 7.0D` is hardcoded. No version negotiation or schema evolution is apparent.