This commit is contained in:
2026-04-17 14:55:32 -04:00
commit bc3ac1d4c9
18017 changed files with 4371742 additions and 0 deletions

View File

@@ -0,0 +1,182 @@
---
source_files:
- Common/DTS.Common.Import/Parsers/CSV/AbstractCSVParser.cs
- Common/DTS.Common.Import/Parsers/CSV/Version0CSVTestParser.cs
- Common/DTS.Common.Import/Parsers/CSV/CSVFile.cs
- Common/DTS.Common.Import/Parsers/CSV/Version6CSVTestParser.cs
- Common/DTS.Common.Import/Parsers/CSV/Version3CSVSensorParser.cs
- Common/DTS.Common.Import/Parsers/CSV/Version5CSVTestParser.cs
- Common/DTS.Common.Import/Parsers/CSV/Version4CSVSensorParser.cs
- Common/DTS.Common.Import/Parsers/CSV/CSVGroupImport.cs
- Common/DTS.Common.Import/Parsers/CSV/DTSCSVTestSetupParser.cs
- Common/DTS.Common.Import/Parsers/CSV/DTSCSVSensorsParser.cs
generated_at: "2026-04-17T15:31:35.696653+00:00"
model: "zai-org/GLM-5-FP8"
schema_version: 1
sha256: "82eec24579c17b93"
---
# CSV Import Parsers Documentation
## 1. Purpose
This module provides CSV parsing functionality for importing DTS sensor configurations and test setups. It implements a versioned parser architecture where different CSV format versions (0-6) are handled by specialized parser classes. The module reads CSV files containing sensor metadata, calibration data, group assignments, and test setup configurations, transforming them into domain objects for the DTS data acquisition system. The architecture separates sensor parsing from test setup parsing, with both feeding into a group import system that creates the final test template structure.
---
## 2. Public Interface
### AbstractCSVParser (Abstract Base Class)
| Member | Signature | Description |
|--------|-----------|-------------|
| Version | `public abstract int Version { get; }` | Returns the parser version number this class handles |
| Initialize | `public void Initialize(ICalibrationImport import, ZeroMethodOptions zmOptions, IImportNotification importNotification, bool importCreateDynamicGroups, bool useISOCodeFilterMapping, bool useZeroForUnfiltered)` | Initializes the parser with dependencies and configuration flags |
| ParseVersion | `public abstract void ParseVersion(CSVImportTags.Tags field, string val, ParseParameters pp)` | Parses a specific field value for this parser's version |
### Version0CSVTestParser
| Member | Signature | Description |
|--------|-----------|-------------|
| Version | `public int Version => 0` | Returns version 0 |
| ParseVersion | `public void ParseVersion(CsvReader csvReader, TestSetupImportData tsid)` | Parses version 0 test setup data including PostTriggerSec, PreTriggerSec, RecordingMode, SampleRate, Description, Name, Version, and Tags |
### Version5CSVTestParser
| Member | Signature | Description |
|--------|-----------|-------------|
| Version | `public int Version => 5` | Returns version 5 |
| ParseVersion | `public void ParseVersion(CsvReader csvReader, TestSetupImportData tsid)` | Parses version 5 test setup data including clock configuration (ClockMasterInputType, ClockMasterOutputType, ClockSlaveInputType, ClockSlaveOutputType, ManageClocksOutsideDPMaster, ManageClocksOutsideDPSlave) |
### Version6CSVTestParser
| Member | Signature | Description |
|--------|-----------|-------------|
| Version | `public int Version => 6` | Returns version 6 |
| ParseVersion | `public void ParseVersion(CsvReader csvReader, TestSetupImportData tsid)` | Parses version 6 test setup data including DAS hardware info (DASSerial, DASSampleRate, PTPDomainId, ClockMaster) with multi-row body parsing |
### Version3CSVSensorParser
| Member | Signature | Description |
|--------|-----------|-------------|
| Version | `public override int Version => 3` | Returns version 3 |
| ParseVersion | `public override void ParseVersion(CSVImportTags.Tags field, string sVal, ParseParameters pp)` | Parses GroupName and GroupType fields with duplicate detection and test object validation |
### Version4CSVSensorParser
| Member | Signature | Description |
|--------|-----------|-------------|
| Version | `public override int Version => 4` | Returns version 4 |
| ParseVersion | `public override void ParseVersion(CSVImportTags.Tags field, string sVal, ParseParameters pp)` | Parses DAS assignment (DASSerialNumber, DASChannelIndex), UDP streaming (StreamProfile, UDPAddress, TimeChannelId, DataChannelId, TmNSConfig, IRIGTimeDataPacketIntervalMS, TMATSIntervalMS), UART configuration (BaudRate, DataBits, StopBits, Parity, DataFormat), and user codes (TestUserCode, TestUserChannelName, TestIsoCode, TestIsoChannelName) |
### CSVFile
| Member | Signature | Description |
|--------|-----------|-------------|
| Constructor | `public CSVFile(string filename)` | Creates instance with specified filename |
| IsInUse | `public static bool IsInUse(string filename)` | Returns true if file is locked/in use by another process |
| IsCSVFileForTestSetupImport | `public static bool IsCSVFileForTestSetupImport(string filename, CsvImportOptions csvImportOptions)` | Validates if CSV file is formatted for test setup import (checks for "Version" header) |
| GetCsvVersion | `public static int GetCsvVersion(string filename)` | Extracts and returns the version number from CSV file, returns 0 on error |
| LineCount | `public int LineCount { get; }` | Returns the number of lines in the file |
### CSVGroupImport
| Member | Signature | Description |
|--------|-----------|-------------|
| ParseParameters | `public ParseParameters ParseParameters { get; set; }` | Gets or sets the parse parameters for import |
| CreateGroups | `public Tuple<TestTemplate, List<IGroup>> CreateGroups(List<SensorData> sensors, Dictionary<string, List<TsetSetupImportSensorInfo>> groupSensorLookup, TestTemplate testTemplate, bool createDynamicGroups, List<IGroup> staticGroups, Action<double> setProgress)` | Creates groups and channels from sensor data, populating the test template |
| GetGroupSensorLookup | `public Dictionary<string, List<TsetSetupImportSensorInfo>> GetGroupSensorLookup(List<SensorData> sensors, Dictionary<string, string> sensorGroupNameLookup, Dictionary<string, List<string>> groupNameSensorListLookup)` | Builds lookup dictionary mapping group names to sensor info lists |
### DTSCSVTestSetupParser
| Member | Signature | Description |
|--------|-----------|-------------|
| Constructor | `public DTSCSVTestSetupParser(IImportNotification importNotification, CsvImportOptions csvImportOptions, TestSetupImportData testSetupImportData, IGroupImport groupImport, bool createDynamicGroups)` | Initializes parser with dependencies |
| CreateTestSetup | `public static TestTemplate CreateTestSetup(TestSetupImportData tsid, DASHardware[] allDAS)` | Creates a TestTemplate from parsed import data |
| UpdateDASSampleRate | `public static void UpdateDASSampleRate(TestTemplate t, TestSetupImportData tsid, DASHardware[] allDAS)` | Updates sample rates for DAS hardware in the test template |
| Parse | `public override void Parse(ref ImportObject importObject)` | Main entry point - parses CSV file and populates ImportObject with test setup data |
### DTSCSVSensorsParser
| Member | Signature | Description |
|--------|-----------|-------------|
| Constructor | `public DTSCSVSensorsParser(IImportNotification importNotification, User user, CsvImportOptions csvImportOptions, ICalibrationImport calibrationImport, ZeroMethodOptions zeroMethodOptions)` | Initializes parser with dependencies |
| ImportCreateDynamicGroups | `public bool ImportCreateDynamicGroups { get; set; }` | Flag for dynamic group creation during import |
| UseISOCodeFilterMapping | `public bool UseISOCodeFilterMapping { get; set; }` | Flag for ISO code filter mapping behavior |
| UseZeroForUnfiltered | `public bool UseZeroForUnfiltered { get; set; }` | Flag for using zero values for unfiltered data |
| Parse | `public override void Parse(ref ImportObject importObject)` | Main entry point - parses CSV file and populates ImportObject with sensor and calibration data |
---
## 3. Invariants
1. **Version Header Requirement**: CSV files must have "Version" as the first token in the first row to be recognized as valid for import.
2. **Version-Based Parser Selection**: The `CSVImportTags.GetVersionForTag()` determines which parser handles each field. Parsers only process fields matching their version.
3. **Version 6 Multi-Parser Execution**: When `tsid.Version == 6`, all parsers from index 1 through the array length are invoked sequentially (see `DTSCSVTestSetupParser.ParseTestSetup`).
4. **Duplicate Serial Number Detection**: `Version3CSVSensorParser` enforces unique sensor serial numbers within `SensorGroupNameLookup` and `SensorGroupTypeLookup` dictionaries.
5. **Non-Empty Group Names**: Group names must be non-empty and non-whitespace; empty values generate errors.
6. **Test Setup Name Uniqueness**: Import skips test setups with names that already exist in `importObject.TestSetups()`.
7. **Calibration Assignment Order**: Calibrations are sorted and the most recent (last element after sort) is assigned to each sensor.
8. **Analog Sensor Validation**: Analog sensors require non-zero sensitivity and capacity >= 1; violations generate errors.
9. **Excitation Validation**: Full/Half/Quarter bridge sensors require defined excitation voltage (not `Undefined`).
---
## 4. Dependencies
### External Dependencies (Imports)
| Namespace | Purpose |
|-----------|---------|
| `CsvHelper` | CSV file reading via `CsvReader` |
| `DTS.Common.Classes.Sensors` | `SensorData`, `SensorCalibration`, `ParseParameters` |
| `DTS.Common.Enums.Sensors` | Sensor enumerations including `BridgeType`, `SensUnits`, `ExcitationVoltageOptions` |
| `DTS.Common.Import.ImportOptions` | `CsvImportOptions`, `ZeroMethodOptions` |
| `DTS.Common.Import.Interfaces` | `IParseCSVSensor`, `IParseCSVTest`, `ICalibrationImport`, `IImportNotification`, `IGroupImport` |
| `DTS.Common.Import.Factories` | `CSVTestParserFactory`, `CSVSensorParserFactory` |
| `DTS.Common.Import.Parsers` | `ParseVariantBase` base class |
| `DTS.Common.Storage` | `DbOperations` for database access |
| `DTS.SensorDB` | Database operations |
| `DataPROWin7.DataModel.Classes.Hardware` | `DASHardwareList` for hardware enumeration |
| `System.IO.Ports` | `StopBits`, `Parity` enumerations for UART configuration |
| `DTS.Slice.Users` | `User` class for current user context |
| `DTS.Common.Utilities.Logging` | `APILogger` for error logging |
### Consumers (Inferred)
The module implements interfaces (`IParseCSVSensor`, `IParseCSVTest`, `IGroupImport`) and extends `ParseVariantBase`, suggesting consumption via:
- Factory patterns (`CSVTestParserFactory`, `CSVSensorParserFactory`)
- Dependency injection through interface types
- The `ImportObject` pattern which aggregates parsed results
---
## 5. Gotchas
1. **Version 6 Special Execution Path**: Unlike other versions, version 6 triggers iteration through all parsers starting at index 1:
```csharp
if (tsid.Version == 6)
{
for (var i = 1; i < parsers.Length; i++)
{
parsers[i].ParseVersion(parser, tsid);
}
}
```
This means version 6 files are processed by multiple parsers, not just `Version6CSVTestParser`.
2. **Silent Sensitivity Unit Correction**: Non-proportional sensors with `SensUnits.mVperVperEU` are silently corrected to `SensUnits.mVperEU`. Users are notified via `_importNotification.ReportErrors` but parsing continues.
3. **ISO Channel-Only Prefix Bypass**: Sensors with serial numbers starting with `Constants.ISO_CH_ONLY_PREFIX` are excluded from `AddSensorLookup` and `AddCalibrationLookup` calls, but still added via `AddSensor` and `AddCalibration`.
4. **Exception Swallowing in CSVFile**:
- `IsInUse()` suppresses all non-IOException exceptions
- `IsCSVFileForTestSetupImport()` and `GetCsvVersion()` catch all exceptions and return `false`/`0`
5. **Generic Exception in Version3CSVSensorParser**: Throws `new Exception("Parse error")` when a group maps to multiple test objects and `ImportCreateDynamicGroups` is false. This breaks the pattern of adding errors to `pp.Errors`.
6. **Dual Sample Rate Storage**: The comment in `UpdateDASSampleRate` explicitly notes uncertainty about why sample rates are stored in both `t.SetSampleRateForHardware()` and `t.DASSampleRateList[]`.
7. **Historical Bug References**: Comments reference "FB" tracking numbers (e.g., "FB 40598", "FB 36879", "FB 43815") indicating accumulated fixes. These may indicate areas with historical issues.
8. **Range Assignment Complexity**: The range assignment logic in `CSVGroupImport.CreateGroups` has specific conditions involving `ImportContainedSensorRanges`, `RangeHigh`, `RangeLow`, `RangeMedium`, and `Capacity` that may produce unexpected results if not fully understood.
9. **Version 6 Header Search Loop**: `Version6CSVTestParser` and `Version5CSVTestParser` both skip rows until finding a version-appropriate tag, but use different version checks (6 vs 5). This could cause confusion if tags overlap.

View File

@@ -0,0 +1,175 @@
---
source_files:
- Common/DTS.Common.Import/Parsers/EQX/EQXTestSetupParser.cs
- Common/DTS.Common.Import/Parsers/EQX/EQXGroupImport.cs
- Common/DTS.Common.Import/Parsers/EQX/EQXSensorsParser.cs
generated_at: "2026-04-17T15:39:25.870361+00:00"
model: "zai-org/GLM-5-FP8"
schema_version: 1
sha256: "605a991ac1639306"
---
# EQX Import Parsers Documentation
## 1. Purpose
This module provides parsing functionality for importing Equipment Exchange (EQX) format files into the DTS system. It consists of three main components: `EQXTestSetupParser` for importing test setup configurations including groups and channel assignments, `EQXSensorsParser` for importing sensor definitions and calibration data, and `EQXGroupImport` for processing group-to-sensor mappings during test setup creation. The module bridges the proprietary EQX XML format with the internal DTS data model, handling version compatibility, calibration assignments, and group hierarchy construction.
---
## 2. Public Interface
### EQXTestSetupParser
**Constructor:**
```csharp
public EQXTestSetupParser(
IImportNotification importNotification,
EqxImportOptions eqxImportOptions,
IGroupImport groupImport,
EquipmentExchange.EQXSensorDatabase eqxSensorDatabase,
bool createDynamicGroups)
```
Initializes the parser with notification services, import options, group import handler, sensor database reference, and a flag indicating whether dynamic groups should be created.
**Methods:**
```csharp
public override void Parse(ref ImportObject importObject)
```
Parses the EQX file specified by `FileName` (inherited from `ParseVariantBase`) and populates the `importObject` with test setup data. Validates that group name test object lookup exists; if not, adds a critical error and aborts. Assigns group lookups, creates test templates, fixes calibrations, assigns groups to test setup, and sets the `TestSetupImportFileFormat` on the import object.
---
### EQXGroupImport
**Implements:** `IGroupImport`
**Properties:**
```csharp
public ParseParameters ParseParameters { get; set; }
```
**Methods:**
```csharp
public Tuple<TestTemplate, List<IGroup>> CreateGroups(
List<SensorData> sensors,
Dictionary<string, List<TsetSetupImportSensorInfo>> groupSensorLookup,
TestTemplate testTemplate,
bool createDynamicGroups,
List<IGroup> staticGroups,
Action<double> setProgress)
```
Creates groups from the provided sensor data and group-sensor lookup. Returns a tuple containing the populated `TestTemplate` and list of static groups. Returns `null` if `groupSensorLookup` is null or empty. Groups are sorted by sensor count (descending), then by name (ascending). Each group is populated with `GroupChannel` objects containing sensor data, calibration info, squib settings, and digital I/O configuration.
```csharp
public Dictionary<string, List<TsetSetupImportSensorInfo>> GetGroupSensorLookup(
List<SensorData> sensors,
Dictionary<string, string> sensorGroupNameLookup,
Dictionary<string, List<string>> groupNameSensorListLookup)
```
Builds a lookup dictionary mapping group names to lists of `TsetSetupImportSensorInfo`. Handles two mutually exclusive input formats: `sensorGroupNameLookup` (sensor → group name) or `groupNameSensorListLookup` (group name → list of sensor serial numbers).
---
### EQXSensorsParser
**Constructor:**
```csharp
public EQXSensorsParser(
IImportNotification importNotification,
User user,
EqxImportOptions eqxImportOptions,
EquipmentExchange.EQXSensorDatabase eqxSensorDatabase)
```
**Constants:**
```csharp
const float MAX_EQX_VERSION_SUPPORT = 1.5F;
```
**Properties:**
```csharp
public bool ImportCreateDynamicGroups { get; set; }
public bool EQXUseSerialNumberFieldForSN { get; set; }
public bool UseZeroForUnfiltered { get; set; }
```
**Methods:**
```csharp
public override void Parse(ref ImportObject importObject)
```
Parses the EQX file for sensor data. Validates XML structure and EQX format version (must be ≤ 1.5). Delegates to `ParseSensor` for actual processing.
---
## 3. Invariants
- **EQX Version Compatibility:** The maximum supported EQX DataFormatEdition is `1.5F`. Files with higher versions will cause import rejection with an error message.
- **Null ImportObject Handling:** Both `EQXTestSetupParser.Parse()` and `EQXSensorsParser.Parse()` throw `ArgumentNullException` if `importObject` is null.
- **Empty FileName Handling:** Both parsers return immediately without processing if `FileName` is null or empty.
- **Group Lookup Mutual Exclusivity:** In `EQXGroupImport.GetGroupSensorLookup()`, either `sensorGroupNameLookup` OR `groupNameSensorListLookup` will be populated, but not both (per comment FB 30358).
- **Calibration Assignment:** In `FixCalibrations()`, calibrations are sorted and the first calibration (index 0) is assigned to each sensor.
- **Test Setup Uniqueness:** `EQXTestSetupParser.Parse()` returns early without creating a test setup if a test setup with the same name already exists in `importObject.TestSetups()`.
- **Group Sorting:** Groups are sorted by sensor count descending, then alphabetically by name ascending when equal.
- **Channel Order:** `GroupChannelOrder` and `TestSetupOrder` are 1-indexed (1 + sensor index).
---
## 4. Dependencies
### Direct Dependencies (Imports):
**EQXTestSetupParser:**
- `DataPROWin7.DataModel` - `ImportObject`, `ImportError`, `ImportSeverityError`, `TestTemplate`, `SensorData`
- `DTS.Common.Import.ImportOptions` - `EqxImportOptions`
- `DTS.Common.Import.Parsers` - `ParseVariantBase`
- `DTS.Common.Interface.Groups.GroupList` - `IGroupImport`, `IGroup`
- `DTS.Common.SharedResource.Strings` - `StringResources`
- `DTS.Common.Utilities.Logging` - `APILogger`
- `DTS.Common.Utils` - `GroupHelper`
- `DTS.SensorDB` - `EquipmentExchange.EQXSensorDatabase`
- `System.Xml.Linq` - XML parsing
**EQXGroupImport:**
- `DataPROWin7.DataModel` - `SensorData`, `TestTemplate`, `TsetSetupImportSensorInfo`
- `DTS.Common.Classes.Groups` - Group-related classes
- `DTS.Common.Classes.Sensors` - `GroupChannel`
- `DTS.Common.Interface.Groups.GroupList` - `IGroupImport`, `IGroup`
- `DTS.Common.Storage` - `DbOperations`
- `DTS.SensorDB` - Sensor database types
**EQXSensorsParser:**
- `DataPROWin7.DataModel` - `ImportObject`, `SensorData`
- `DTS.Common.Enums` - Enumerations including `BridgeType`, `ExcitationVoltageOptions`, `SquibMeasurementType`
- `DTS.Common.Import.ImportOptions` - `EqxImportOptions`
- `DTS.Common.Import.Parsers` - `ParseVariantBase`
- `DTS.Common.Storage` - `DbOperations`
- `DTS.Common.Classes.Sensors` - `FactorySensorModel`, `SensorsCollection`
- `DTS.Common.Interface.Sensors` - `ISensorData`
- `DTS.Common.SharedResource.Strings` - `StringResources`
- `DTS.Slice.Users` - `User`
- `DTS.SensorDB` - `EquipmentExchange.EQXSensorDatabase`
- `System.Xml.Linq` - XML parsing
### Consumers:
- The module is consumed by the DTS import system (exact consumers not visible in provided source).
---
## 5. Gotchas
### Calibration Reset Issue (FB 44105 / Historical Bug)
The comment in `EQXTestSetupParser.AssignGroupsToTestSetup()` references a bug where `GroupHelper.NormalizeSensorIds()` can reset calibrations for sensors. The workaround is calling `FixCalibrations(importObject)` immediately after normalization to restore calibrations from the import data.
### Duplicate Calibration Fix Calls
`FixCalibrations()` is called twice in `EQXTestSetupParser.Parse()`: once early in the method, and again inside `AssignGroupsToTestSetup()` after `NormalizeSensorIds()`. This is intentional due to the normalization side-effect noted above.
### EID Preservation (Bug 18467)
`EQXSensorsParser.ParseSensor()` contains special logic to preserve existing EID values when the EQX file has a NULL `IDModuleString`. Without this, importing an EQX file with missing IDModuleString would clear the EID on existing sensors.
### Null IDModule Handling
The method `SensorHasNullIDModule()` and `GetSensorNullIdModuleValue()` are called on `_eq