DP44/enriched-qwen3-coder-next/Common/DTS.Common.Serialization/HDF.md

source_files, generated_at, model, schema_version, sha256

source_files	generated_at	model	schema_version	sha256
Common/DTS.Common.Serialization/HDF/HDF.File.cs Common/DTS.Common.Serialization/HDF/HDF.File.Writer.cs	2026-04-16T03:36:32.708668+00:00	Qwen/Qwen3-Coder-Next-FP8	1	6a62f69116272309

HDF

Documentation: DTS.Serialization.HDF.File

---

1. Purpose

This module implements a concrete serialization backend for writing test data to HDF5 files, specifically tailored for DTS’s data acquisition systems. It extends Serialization.File and implements IWritable<Test>, enabling structured export of test metadata, raw sensor data (in ADC, mV, and Engineering Units), and associated binary files (logs, reports, setup, .dts) into a hierarchical HDF5 container. The implementation is tightly coupled to DTS’s internal data models (Test, AnalogInputChannel, DataScaler) and supports WIAMan-compliant metadata and S6 system attributes for interoperability with external analysis tools.

---

2. Public Interface

public partial class File : Serialization.File, IWritable<Test>

• File()
  Constructor. Initializes the base class with "HDF" as the file type identifier.

• IWriter<Test> Exporter { get; }
  Lazily instantiates and returns a Writer instance, using the default encoding. Throws an Exception wrapped around any inner exception encountered during writer creation.

• bool ExportADC { set; }
  Controls whether raw ADC data is exported. Delegates to ((Writer)Exporter).ExportADC.

• bool ExportEU { set; }
  Controls whether engineering unit (EU) data is exported. Delegates to ((Writer)Exporter).ExportEU.

• bool ExportMV { set; }
  Controls whether millivolt (mV) data is exported. Delegates to ((Writer)Exporter).ExportMV.

• bool ExportLogs { set; }
  Controls whether log files (from /Logs directory) are included in the HDF5 /Files/Logs group. Delegates to ((Writer)Exporter).ExportLogs.

• bool ExportReports { set; }
  Controls whether report files (from /Reports directory) are included in /Files/Reports. Delegates to ((Writer)Exporter).ExportReports.

• bool ExportSetup { set; }
  Controls whether setup files (from /SETUP directory) are included in /Files/SETUP. Delegates to ((Writer)Exporter).ExportSetup.

• bool ExportDTSFile { set; }
  Controls whether .dts files (from /Binary/{dataDir}) are included in /Files/Binary/{dataDir}. Delegates to ((Writer)Exporter).ExportDTSFile.

• string CustomerName { set; }
  Sets WIAMan metadata attribute Director. Delegates to ((Writer)Exporter).CustomerName.

• string TestEngineerName { set; }
  Sets WIAMan metadata attribute Operator. Delegates to ((Writer)Exporter).TestEngineerName.

• string LabName { set; }
  Sets WIAMan metadata attribute Location. Delegates to ((Writer)Exporter).LabName.

• bool IsWiamanData { set; }
  Sets WIAMan metadata attribute Is WIAMan Data. Delegates to ((Writer)Exporter).IsWiamanData.

• Dictionary<string, string> ISOToFineLocation3 { set; }
  Used to populate WIAMan attribute Anthropomorphic Label. Delegates to ((Writer)Exporter).ISOToFineLocation3.

• Dictionary<string, string> ISOToPhysicalDimension { set; }
  Used to populate WIAMan attribute Channel Label:Category. Delegates to ((Writer)Exporter).ISOToPhysicalDimension.

• Dictionary<string, string> ISOToPosition { set; }
  Used to populate WIAMan attribute Channel Label:Optional. Delegates to ((Writer)Exporter).ISOToPosition.

• Dictionary<string, string> ISOToTransducerMainLocation { set; }
  Not used in the current implementation. Delegates to ((Writer)Exporter).ISOToTransducerMainLocation, but the setter is present in File while the corresponding property is not declared in Writer (see Gotchas).

---

public class Writer : Writer<File>, IWriter<Test>

• internal File WriterParent { get; }
  Reference to the owning File instance.

• bool ExportADC { get; set; }
  Controls ADC data export. Default: true.

• bool ExportEU { get; set; }
  Controls EU data export. Default: true.

• bool ExportMV { get; set; }
  Controls mV data export. Default: true.

• bool ExportLogs { get; set; }
  Controls log file inclusion. Default: true.

• bool ExportReports { get; set; }
  Controls report file inclusion. Default: true.

• bool ExportSetup { get; set; }
  Controls setup file inclusion. Default: true.

• bool ExportDTSFile { get; set; }
  Controls .dts file inclusion. Default: true.

• string CustomerName { get; set; }
  WIAMan metadata: Director.

• string TestEngineerName { get; set; }
  WIAMan metadata: Operator.

• string LabName { get; set; }
  WIAMan metadata: Location.

• bool IsWiamanData { get; set; }
  WIAMan metadata: Is WIAMan Data.

• Dictionary<string, string> ISOToFineLocation3 { get; set; }
  Used to construct Anthropomorphic Label.

• Dictionary<string, string> ISOToPhysicalDimension { get; set; }
  Used to construct Channel Label:Category.

• Dictionary<string, string> ISOToPosition { get; set; }
  Used to construct Channel Label:Optional.

• Dictionary<string, string> ISOToTransducerMainLocation { get; set; }
  Declared in Writer but never used. (See Gotchas.)

• void Write(string pathname, string id, string dataFolder, Test test, bool bFiltering, bool includeGroupNameInISOExport, FilteredData fd, Test.Module.Channel tmChannel, int channelNumber, BeginEventHandler beginEventHandler, CancelEventHandler cancelEventHandler, EndEventHandler endEventHandler, TickEventHandler tickEventHandler, ErrorEventHandler errorEventHandler, CancelRequested cancelRequested, double minStartTime, int dataCollectionLength)
  Main export method. Writes the Test object to an HDF5 file at pathname.

  • Creates /Files/{folder} groups for logs, setup, reports, and .dts files if Export* flags are enabled.
  • Writes three dataset groups: /Datasets_EU, /Datasets, /Datasets_MV, depending on flags.
  • For each AnalogInputChannel, writes channel-specific metadata (WIAMan + S6 attributes) and raw data as datasets named {index}: Strain_RawYData.
  • Uses Parallel.ForEach over test.Channels for performance.
  • Progress is reported via tickEventHandler in 100-step increments.
  • Errors are logged via APILogger and propagated via errorEventHandler.

• private Common.DAS.Concepts.DataScaler GetDataScaler(Test.Module.AnalogInputChannel currentAnalogChannel)
  Constructs a DataScaler from channel properties (e.g., UnitConversion, Multiplier, Sensitivity, ZeroMethod, RemovedADC, MeasuredExcitationVoltage, etc.). Includes robust error handling with logging for individual scaler property assignments.

• private void CheckStatus(long status, ErrorLocation location)
  Throws an Exception with location.ToString() if status < 0. Used after HDF5 API calls.

• private void CreateStringAttribute(long objectId, string attributeName, string attributeValue)
  Creates an HDF5 string attribute on objectId. Handles null/empty values gracefully. Uses H5S.create, H5T.copy, H5A.create, H5A.write, and Marshal.StringToHGlobalAnsi. All handles are closed in finally.

• private void CreateIntAttribute(long objectId, string title, int value)
  If ATTRIBUTE_STRING_DATATYPE_ONLY is true (hardcoded), converts value to string and delegates to CreateStringAttribute. Otherwise, would create numeric attribute (not implemented).

• private void CreateUlongAttribute(long objectId, string title, ulong value)
  Same behavior as CreateIntAttribute.

• private void CreateDoubleAttribute(long objectId, string title, double value)
  Same behavior as CreateIntAttribute.

• private void AddDirectoryIfExists(string binaryPath, string folder, long hdfObjectId, string fileExtension)
  Adds all files matching fileExtension from a computed source directory (e.g., root/Logs) into an HDF5 group at /Files/{folder}. Uses nested DirectoryInfo.Parent checks to determine root. Reads files into byte arrays and writes them as 1D NATIVE_UCHAR datasets.

• private int ComputeNumberOfSteps(Test test)
  Computes total export steps based on enabled export flags and channel count. Each enabled export type contributes Channels.Count steps; each file group (logs, setup, reports, dts) contributes 1 step.

• internal Writer(File fileType, int encoding)
  Constructor. Initializes all Export* flags to true. Sets WriterParent.

• public void Initialize(...)
  Empty implementation. No initialization logic is performed.

---

3. Invariants

• HDF5 file structure:

  • /Files/{folder} groups are created only if corresponding Export* flag is true.
  • /Datasets, /Datasets_EU, /Datasets_MV groups are created only if ExportADC, ExportEU, or ExportMV is true, respectively.
  • Each channel’s data resides under a group named /{datasetGroup}/{index:0000}: {channelName}.
  • Channel data datasets are named {index}: Strain_RawYData.

• Data type in datasets:

  • ADC: H5T.NATIVE_SHORT (16-bit signed integer).
  • mV/EU: H5T.NATIVE_DOUBLE (64-bit float).

• Channel ordering:
  Channels are sorted by AbsoluteDisplayOrder before processing.

• Progress reporting:
  Progress updates occur every UPDATE_INTERVAL = 1000 samples per channel, and at group-level milestones (e.g., after each file group or dataset group is written). Final progress is always set to 100D in finally.

• WIAMan attribute requirement:
  All WIAMan attributes listed in code comments are written only if IsWiamanData is true?
  → No: Attributes are written unconditionally. IsWiamanData only sets a string attribute "Is WIAMan Data" to "TRUE"/"FALSE".

• String attributes:
  All attributes (int, double, ulong) are stored as strings if ATTRIBUTE_STRING_DATATYPE_ONLY = true (hardcoded). No numeric HDF5 attributes are created.

• Locking:
  HDF5 API calls are guarded by H5WriteLock (a private object). Progress updates use _updateProgressLock.

---

4. Dependencies

Imports/Usings:

• HDF.PInvoke — P/Invoke wrapper for HDF5 C library.
• System.Runtime.InteropServices — For Marshal.AllocHGlobal, FreeHGlobal, StringToHGlobalAnsi.
• DTS.Common.Enums.Sensors — For SensorConstants.BridgeType.
• DTS.Common.Utilities.Logging — For APILogger.
• System.Collections.Generic, System.IO, System.Linq, System.Threading.Tasks — Standard .NET.

Internal Dependencies:

• Serialization.File — Base class (inherited).
• IWriter<Test> — Interface implemented.
• Test — Core data model (from DTS.Serialization or similar).
• Test.Module, Test.Module.AnalogInputChannel, FilteredData, DataScaler — Internal DTS types.
• BeginEventHandler, CancelEventHandler, EndEventHandler, TickEventHandler, ErrorEventHandler, CancelRequested — Delegates for progress/cancellation.

External Dependencies:

• HDF5 library (via HDF.PInvoke).
• File system access for reading logs, setup, reports, and .dts files.

---

5. Gotchas

• ISOToTransducerMainLocation is unused:
  The File class exposes a setter for ISOToTransducerMainLocation, and the Writer class declares a corresponding property, but the property is never referenced in the code. This is likely dead code.

• Initialize method is empty:
  The Writer.Initialize(...) method has no implementation. All work is done in Write(...). This may violate expectations of a two-phase initialization pattern.

• String-only attributes:
  All numeric attributes (int, double, ulong) are stored as strings due to ATTRIBUTE_STRING_DATATYPE_ONLY = true. This increases file size and complicates downstream parsing.

• Nested DirectoryInfo.Parent checks:
  The AddDirectoryIfExists method uses a brittle chain of Parent.Parent.Parent to determine the root directory. This assumes a fixed directory structure and may fail if the path depth is insufficient.

• Parallel channel processing:
  Channels are processed in parallel (Parallel.ForEach), but progress updates are not thread-safe beyond the _updateProgressLock. Steps completed (stepsCompleted) is incremented outside the lock, risking race conditions and incorrect progress reporting.

• Hardcoded dataset names:
  All channel datasets are named {index}: Strain_RawYData, regardless of data type (ADC/mV/EU). This may cause confusion or overwriting if multiple dataset groups exist.

• No cancellation support in Write:
  Although cancelRequested is passed to Write, it is never checked. The export runs to completion regardless of cancellation.

• Error handling is lossy:
  CheckStatus throws a generic Exception with only the location name (e.g., "File"), losing the underlying HDF5 error code or message. Detailed diagnostics rely on APILogger.Log calls.

• Memory allocation per sample:
  Data is written sample-by-sample using Marshal.WriteByte in loops. This is inefficient compared to bulk copy (e.g., Marshal.Copy). The comment references performance fixes, but the implementation remains suboptimal.

• No validation of ISOTo* dictionaries:
  Accessing ISOToFineLocation3[aic.IsoCode] without checking for key existence will throw KeyNotFoundException if the dictionary is incomplete or keys are missing.

• minStartTime and dataCollectionLength are unused:
  These parameters appear in Write signatures but are never used in the implementation.

• ExtensionPrefix is settable but unused:
  The ExtensionPrefix property is defined but not used in filename construction ({id}_{LabName}_1of1{(ExtensionPrefix ?? "")}.h5), so it only affects the filename if explicitly set to a non-empty value.

• No support for digital or calculated channels:
  Parallel.ForEach skips non-AnalogInputChannel instances (if (null == aic) return;). Digital channels, calculated channels, or other channel types are silently ignored.

• Time handling assumptions:
  Reference Time is computed from TriggerTimestampSec and TriggerTimestampNanoSec, assuming Unix epoch. No timezone or leap-second handling is evident.

• Hardcoded export version:
  HDF_EXPORT_VERSION = 7.0D is hardcoded. No version negotiation or schema evolution is apparent.