noisedestroyers/DP44
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

init

Browse Source
This commit is contained in:
noisedestroyers
2026-04-17 14:55:32 -04:00
commit bc3ac1d4c9
18017 changed files with 4371742 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

143
enriched-qwen3-coder-next/Common/DTS.Common.Import/Parsers/CSV.md Normal file
View File

@@ -0,0 +1,143 @@
---
source_files:
- Common/DTS.Common.Import/Parsers/CSV/AbstractCSVParser.cs
- Common/DTS.Common.Import/Parsers/CSV/Version0CSVTestParser.cs
- Common/DTS.Common.Import/Parsers/CSV/CSVFile.cs
- Common/DTS.Common.Import/Parsers/CSV/Version6CSVTestParser.cs
- Common/DTS.Common.Import/Parsers/CSV/Version3CSVSensorParser.cs
- Common/DTS.Common.Import/Parsers/CSV/Version5CSVTestParser.cs
- Common/DTS.Common.Import/Parsers/CSV/Version4CSVSensorParser.cs
- Common/DTS.Common.Import/Parsers/CSV/CSVGroupImport.cs
- Common/DTS.Common.Import/Parsers/CSV/DTSCSVTestSetupParser.cs
- Common/DTS.Common.Import/Parsers/CSV/DTSCSVSensorsParser.cs
generated_at: "2026-04-16T02:08:24.981291+00:00"
model: "Qwen/Qwen3-Coder-Next-FP8"
schema_version: 1
sha256: "2bff5b23b8f73415"
---
# CSV Import Module Documentation
## 1. Purpose
This module provides a versioned CSV parsing infrastructure for importing sensor and test setup data into the DTS system. It supports two distinct import workflows: **sensor data import** (handled by `DTSCSVSensorsParser`) and **test setup data import** (handled by `DTSCSVTestSetupParser`). The module uses an abstract base class (`AbstractCSVParser`) and concrete version-specific parsers to handle evolving CSV formats across versions 0 through 6. It decouples parsing logic from business logic, delegates group creation to `CSVGroupImport`, and integrates with calibration, sensor, and hardware data models to construct `TestTemplate` and `ImportObject` instances for downstream processing.
## 2. Public Interface
### `AbstractCSVParser` (abstract base class)
- **`int Version { get; }`**
Returns the CSV format version supported by this parser (e.g., 3, 4).
- **`void Initialize(ICalibrationImport import, ZeroMethodOptions zmOptions, IImportNotification importNotification, bool importCreateDynamicGroups, bool useISOCodeFilterMapping, bool useZeroForUnfiltered)`**
Injects dependencies and configuration flags required for parsing. Must be called before `ParseVersion`.
- **`abstract void ParseVersion(CSVImportTags.Tags field, string val, ParseParameters pp)`**
Parses a single field (`field`) from a sensor CSV row, using value `val`. Errors are appended to `pp.Errors`. Behavior is version-specific.
### `Version0CSVTestParser`
- **`int Version => 0`**
- **`void ParseVersion(CsvReader csvReader, TestSetupImportData tsid)`**
Parses test setup metadata (e.g., `PostTriggerSec`, `SampleRate`, `RecordingMode`) from a CSV with a simple header-value pair format. Reads two rows: header names and values.
### `Version3CSVSensorParser`
- **`int Version => 3`**
- **`void ParseVersion(CSVImportTags.Tags field, string sVal, ParseParameters pp)`**
Handles `GroupName` and `GroupType` fields. Enforces non-empty values, uniqueness of sensor serial numbers in group lookups, and consistency of `TestObject` per group (throws `Exception("Parse error")` on conflict if `ImportCreateDynamicGroups` is false).
### `Version4CSVSensorParser`
- **`int Version => 4`**
- **`void ParseVersion(CSVImportTags.Tags field, string sVal, ParseParameters pp)`**
Parses DAS-related (`DASSerialNumber`, `DASChannelIndex`), streaming (`StreamProfile`, `UDPAddress`, `TimeChannelId`, `DataChannelId`, `TmNSConfig`, `IRIGTimeDataPacketIntervalMS`, `TMATSIntervalMS`), UART (`BaudRate`, `DataBits`, `StopBits`, `Parity`, `DataFormat`), and user-defined fields (`TestUserCode`, `TestUserChannelName`, `TestIsoCode`, `TestIsoChannelName`). Updates `pp.SensorData` and lookup dictionaries.
### `Version5CSVTestParser`
- **`int Version => 5`**
- **`void ParseVersion(CsvReader csvReader, TestSetupImportData tsid)`**
Parses clock synchronization settings (e.g., `ClockMasterInputType`, `ClockMasterOutputType`, `ManageClocksOutsideDPMaster`) from a header-value row format.
### `Version6CSVTestParser`
- **`int Version => 6`**
- **`void ParseVersion(CsvReader csvReader, TestSetupImportData tsid)`**
Parses per-DAS hardware configuration (e.g., `DASSerial`, `DASSampleRate`, `PTPDomainId`, `ClockMaster`) in a table format. Reads header row, then iterates rows until an empty line or non-matching field is encountered.
### `CSVFile`
- **`static bool IsInUse(string filename)`**
Returns `true` if the file is locked (IOException on `File.ReadAllLines`), else `false`.
- **`static bool IsCSVFileForTestSetupImport(string filename, CsvImportOptions csvImportOptions)`**
Returns `true` if the first field of the first row is `"Version"`.
- **`static int GetCsvVersion(string filename)`**
Returns the integer value in the first column of the second row if the first row starts with `"Version"`, else `0`.
- **`int LineCount { get; }`**
Returns the number of lines in the file (0 on error or null filename).
### `CSVGroupImport`
- **`ParseParameters ParseParameters { get; set; }`**
- **`Tuple<TestTemplate, List<IGroup>> CreateGroups(List<SensorData> sensors, Dictionary<string, List<TsetSetupImportSensorInfo>> groupSensorLookup, TestTemplate testTemplate, bool createDynamicGroups, List<IGroup> staticGroups, Action<double> setProgress)`**
Creates `IGroup` instances from `groupSensorLookup`, assigns sensor/channel properties (ISO/user codes, DAS mapping, calibration defaults), and adds groups to `testTemplate`. Updates `setProgress`.
- **`Dictionary<string, List<TsetSetupImportSensorInfo>> GetGroupSensorLookup(List<SensorData> sensors, Dictionary<string, string> sensorGroupNameLookup, Dictionary<string, List<string>> groupNameSensorListLookup)`**
Builds a mapping from group name to list of `TsetSetupImportSensorInfo` (containing sensor/DAS parameters) using either `sensorGroupNameLookup` or `groupNameSensorListLookup`.
### `DTSCSVTestSetupParser`
- **`void Parse(ref ImportObject importObject)`**
Orchestrates test setup import: validates CSV format, parses test setup data via `ParseTestSetup`, creates `TestTemplate`, assigns hardware, clock sync, and tags, then delegates group creation to `AssignGroupsToTestSetup`. Adds result to `importObject`.
### `DTSCSVSensorsParser`
- **`void Parse(ref ImportObject importObject)`**
Orchestrates sensor import: reads CSV, populates `SensorData` and `SensorCalibration` objects via `PopulateSensor`, handles sensitivity unit corrections, and adds sensors/calibrations to `importObject`. Reports errors and warnings.
## 3. Invariants
- **CSV Format Validation**:
- A CSV is recognized as a test setup file if its first field is `"Version"`.
- Sensor CSVs must have recognizable column headers (else `ImportSensorsPreviewControl_NoColumnsInTDCCSV` error).
- **Version Consistency**:
- `Version0CSVTestParser` and `Version5CSVTestParser` expect header-value pairs.
- `Version6CSVTestParser` expects a table format with header row followed by data rows until an empty line.
- `DTSCSVTestSetupParser.ParseTestSetup` uses `CSVFile.GetCsvVersion` to determine which parsers to invoke; version 6 triggers all parsers (0 + 5 + 6).
- **Sensor Grouping**:
- `Version3CSVSensorParser` enforces that a group name maps to a single `SensorTestObject` unless `ImportCreateDynamicGroups` is true.
- Duplicate sensor serial numbers in `sensorGroupNameLookup` or `sensorGroupTypeLookup` are rejected with errors.
- **Calibration & Defaults**:
- `DTSCSVSensorsParser` applies user-specific defaults for squib/digital output settings.
- Sensitivity units are corrected from `mVperVperEU` to `mVperEU` for non-proportional calibrations.
- Zero method is initialized from `pp.SensorCal.ZeroMethods.Methods[0]` during `PostParse`.
- **Hardware Mapping**:
- DAS hardware is added to `TestTemplate` only if version ≥ 6 (per `DTSCSVTestSetupParser` comment FB 43815).
- DAS sample rates are set in both `t.SetSampleRateForHardware` and `t.DASSampleRateList`.
## 4. Dependencies
### Imports/References:
- **Core Types**: `DTS.Common.Classes`, `DTS.Common.Enums`, `DTS.Common.Import.Interfaces`, `CsvHelper`, `System.IO`, `System.Collections.Generic`.
- **Hardware/Storage**: `DataPROWin7.DataModel`, `DTS.SensorDB`, `DTS.Common.Storage`.
- **Import Infrastructure**: `ParseVariantBase`, `ParseParameters`, `ImportObject`, `TestSetupImportData`, `SensorData`, `SensorCalibration`, `IGroup`, `IGroupImport`, `ICalibrationImport`, `IImportNotification`.
- **CSV Tags**: `CSVImportTags` (static class providing `GetTagForString`, `GetVersionForTag`).
- **Factories**: `CSVTestParserFactory`, `CSVSensorParserFactory`.
### Used By:
- `DTSCSVTestSetupParser` is invoked by the test setup import pipeline.
- `DTSCSVSensorsParser` is invoked by the sensor import pipeline.
- `CSVGroupImport` is used by `DTSCSVTestSetupParser` to create groups.
## 5. Gotchas
- **Version 6 CSV Parsing Quirk**:
In `DTSCSVSensorsParser.ParseSensor`, if version is 6, the parser skips rows until it finds a tag with version ≤ 4 (to avoid parsing test setup-specific rows as sensor data). This assumes sensor and test setup data are in the same file but separated by an empty line.
- **Exception Thrown in Group Parsing**:
`Version3CSVSensorParser.ParseVersion` throws `new Exception("Parse error")` on group/test object conflict (if `ImportCreateDynamicGroups` is false). This is caught by `DTSCSVSensorsParser` and reported as a sensor error, but the exception itself is non-descriptive.
- **Sensitivity Unit Correction**:
`DTSCSVSensorsParser` silently changes `SensitivityUnits` from `mVperVperEU` to `mVperEU` for non-proportional calibrations. This is logged as a warning but may cause unexpected calibration behavior.
- **DAS Hardware Lookup**:
`CSVGroupImport.CreateGroups` uses `DbOperations.GetChannelSettingDefaults()` and `GroupHelper.GetDAS` to map DAS serial numbers to IDs. If DAS is not in `DASHardwareList.GetAllHardware()`, channel DAS ID remains unset.
- **Zero Method Initialization**:
`PostParse` in `DTSCSVSensorsParser` initializes `pp.ZeroType`, `pp.ZeroStart`, `pp.ZeroEnd` from `pp.SensorCal.ZeroMethods.Methods[0]`. If `ZeroMethods` is uninitialized or empty, this may cause `IndexOutOfRangeException`.
- **No Validation of Sensor Serial Number Format**:
Sensors with serial numbers starting with `Constants.ISO_CH_ONLY_PREFIX` bypass some validation (e.g., sensitivity/capacity checks), but the prefix is not defined in the provided source.
- **Error Reporting Inconsistency**:
`DTSCSVSensorsParser` aggregates errors per line and reports them as a single error string, while `DTSCSVTestSetupParser` reports errors via `IImportNotification.ReportErrors`. Error severity and continuation behavior differ between parsers.
- **File Lock Detection Limitation**:
`CSVFile.IsInUse` uses `File.ReadAllLines`, which may not reliably detect all lock states (e.g., file opened for append-only).

115
enriched-qwen3-coder-next/Common/DTS.Common.Import/Parsers/EQX.md Normal file
View File

@@ -0,0 +1,115 @@
---
source_files:
- Common/DTS.Common.Import/Parsers/EQX/EQXTestSetupParser.cs
- Common/DTS.Common.Import/Parsers/EQX/EQXGroupImport.cs
- Common/DTS.Common.Import/Parsers/EQX/EQXSensorsParser.cs
generated_at: "2026-04-16T02:08:30.752253+00:00"
model: "Qwen/Qwen3-Coder-Next-FP8"
schema_version: 1
sha256: "ce57e1c33a0387ff"
---
# Documentation: EQX Test Setup Parser Module
## 1. Purpose
This module provides parsing logic for EQX files that contain test setup information (i.e., groupings of sensors into test groups and assignment to a test template). It is part of the `DTS.Common.Import` namespace and is specifically responsible for interpreting group-to-sensor mappings defined in an EQX file, constructing `TestTemplate`, static `IGroup`, and `SensorData` objects accordingly, and integrating them into the `ImportObject`. It depends on prior parsing of sensor and calibration data (handled by `EQXSensorsParser`) and uses `EQXGroupImport` to implement group creation logic. The parser aborts the import if the EQX file lacks test setup data (e.g., no `GroupNameTestObjectLookup`).
## 2. Public Interface
### `EQXTestSetupParser` class
**Constructor**
```csharp
public EQXTestSetupParser(
IImportNotification importNotification,
EqxImportOptions eqxImportOptions,
IGroupImport groupImport,
EquipmentExchange.EQXSensorDatabase eqxSensorDatabase,
bool createDynamicGroups)
```
Initializes the parser with dependencies required for parsing and group creation.
- `importNotification`: Used for reporting errors and progress.
- `eqxImportOptions`: Configuration options for EQX import.
- `groupImport`: Implementation of `IGroupImport` (typically `EQXGroupImport`) for group creation logic.
- `eqxSensorDatabase`: Pre-populated sensor database from `EQXSensorsParser`.
- `createDynamicGroups`: Flag indicating whether dynamic groups should be created (if `false`, static groups are created).
**Override Method**
```csharp
public override void Parse(ref ImportObject importObject)
```
Main entry point for parsing test setup data from the EQX file.
- Validates `FileName` and `importObject` (throws `ArgumentNullException` if `importObject` is `null`).
- Checks for presence of test setup data in `sensorImportData.GroupNameTestObjectLookup`; if missing, adds a critical error and aborts.
- Assigns group-to-object and group-to-sensor lookups from the database to `importObject`.
- Extracts setup name from XML using `ParseSetupName`, falls back to file name.
- Skips if a test setup with the same name already exists in `importObject`.
- Calls `FixCalibrations` to ensure sensors use the correct calibration from import.
- Delegates group creation to `AssignGroupsToTestSetup`.
- Cleans sensor data placeholders, clears existing sensors, adds cleaned sensors, and adds the resulting `TestTemplate` and static groups to `importObject`.
- Sets `importObject.TestSetupImportFileFormat`.
**Private Helper Methods**
```csharp
private void FixCalibrations(ImportObject importObject)
```
For each sensor in `importObject`, retrieves calibrations via `CalibrationLookup(sensor.SerialNumber)`, sorts them, and assigns the first (earliest) calibration to `sensor.Calibration`. Logs exceptions via `APILogger`.
```csharp
private Tuple<TestTemplate, List<IGroup>, List<SensorData>> AssignGroupsToTestSetup(
TestTemplate testTemplate,
ImportObject importObject,
Dictionary<string, List<string>> groupNameSensorListLookup,
IGroupImport groupImport,
IImportNotification importNotification)
```
Implements the group assignment logic (moved here per FB 36879).
- Reverses channel order using `GroupHelper.ReverseChannelOrder`.
- Calls `groupImport.GetGroupSensorLookup` to build `groupSensorLookup`.
- Normalizes sensor IDs via `GroupHelper.NormalizeSensorIds`.
- Re-applies `FixCalibrations` (to revert potential calibration resets from normalization).
- Calls `groupImport.CreateGroups(...)` to generate groups and test setup.
- Returns `Tuple<TestTemplate, List<IGroup>, List<SensorData>>`, or `null` on failure (with error reported via `importNotification`).
- Returns `null` for `testTemplate`/`staticGroups` if `groupImport.CreateGroups` returns `null`.
```csharp
private string ParseSetupName(string filename)
```
Parses the `<SetupName>` value from the first `<Fields>` element under `<Sensors>` in the EQX XML file. Returns empty string on any failure (including missing element or XML parse error).
## 3. Invariants
- **`importObject` must be non-null** at start of `Parse`; otherwise, `ArgumentNullException` is thrown.
- **`GroupNameTestObjectLookup` must be non-null and non-empty** in `sensorImportData`; otherwise, a critical error is added and parsing aborts.
- **Calibrations must be sorted before selecting the first one** in `FixCalibrations`; sorting is performed explicitly.
- **Sensors are cleared and re-added** after cleaning (`GroupHelper.CleanUneededSensorDataPlaceHolder`) to ensure consistency.
- **Test setup name uniqueness is enforced**: if a test setup with the same name already exists in `importObject.TestSetups()`, the method returns early without modifying the object.
- **`EQXGroupImport.CreateGroups` may return `null`**, in which case `AssignGroupsToTestSetup` returns a tuple of `null` values and no groups/test setup are added.
## 4. Dependencies
### Dependencies *of* this module:
- **`EQXSensorsParser`**: Must run first to populate `ImportObject.Sensors()`, `ImportObject.CalibrationsLookup()`, and `EquipmentExchange.EQXSensorDatabase` (via `EQXSensorDatabase.Read`).
- **`EquipmentExchange.EQXSensorDatabase`**: Must be pre-populated with sensor, calibration, and group mapping data from the EQX file.
- **`IGroupImport`**: Typically implemented by `EQXGroupImport`; provides `GetGroupSensorLookup` and `CreateGroups`.
- **`GroupHelper`**: Used for `ReverseChannelOrder`, `NormalizeSensorIds`, and `CleanUneededSensorDataPlaceHolder`.
- **`DbOperations`**: Referenced indirectly (e.g., `GetChannelSettingDefaults`, `TagsGet`, etc.) via `EQXGroupImport` and `EQXSensorsParser`.
- **`APILogger`**: Used for logging failures in `FixCalibrations`.
- **`StringResources`**: Used for error messages (e.g., `Import_EQXFileNotForTestSetup`).
### Dependencies *on* this module:
- **`EQXSensorsParser`**: Must be invoked before `EQXTestSetupParser` to ensure sensor and calibration data are available.
- **`ImportObject` consumers**: Any downstream logic that consumes `TestTemplate`, `StaticGroups`, or `Sensors` after import will depend on this parser’s output.
## 5. Gotchas
- **Calibration resets after normalization**: `GroupHelper.NormalizeSensorIds` may reset sensor calibrations; `FixCalibrations` is called *twice* (before and after normalization) to mitigate this (see FB 44105 comment in `AssignGroupsToTestSetup`).
- **Setup name extraction is fragile**: `ParseSetupName` silently returns `string.Empty` on any XML parsing error or missing `<SetupName>` element.
- **No overwrite of existing test setups**: If a test setup with the same name already exists, `Parse` returns early without warning or modification (only silent skip).
- **Group creation may silently fail**: If `groupImport.CreateGroups` throws or returns `null`, `AssignGroupsToTestSetup` returns `null` and no groups/test setup are added, but only the exception message is reported via `importNotification.ReportErrors` (no additional context).
- **`ImportCreateDynamicGroups` flag is not used directly in this class**: It is passed to `EQXGroupImport.CreateGroups` via `EQXTestSetupParser` constructor, but the flag is stored as a property in `EQXSensorsParser`, not `EQXTestSetupParser`.
- **Excitation errors are computed but not reported**: `EQXSensorsParser.GetExcitationErrors()` collects errors but the `TODO Report the errors` comment indicates they are not currently surfaced (only in `EQXSensorsParser`, not here).
- **`SensorData` is modified in-place**: `FixCalibrations` mutates `sensor.Calibration` directly; callers must be aware of side effects.
- **No handling of duplicate sensor serial numbers**: If duplicate serial numbers exist in the import, behavior is undefined (e.g., `sensorLookup` in `EQXGroupImport` will only retain the last one).