noisedestroyers/DP44
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

init

Browse Source
This commit is contained in:
noisedestroyers
2026-04-17 14:55:32 -04:00
commit bc3ac1d4c9
18017 changed files with 4371742 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

108
enriched-partialglm/Common/DTS.Common.Import/Parsers/CSV.md Normal file
View File

@@ -0,0 +1,108 @@
---
source_files:
- Common/DTS.Common.Import/Parsers/CSV/AbstractCSVParser.cs
- Common/DTS.Common.Import/Parsers/CSV/Version0CSVTestParser.cs
- Common/DTS.Common.Import/Parsers/CSV/CSVFile.cs
- Common/DTS.Common.Import/Parsers/CSV/Version6CSVTestParser.cs
- Common/DTS.Common.Import/Parsers/CSV/Version3CSVSensorParser.cs
- Common/DTS.Common.Import/Parsers/CSV/Version5CSVTestParser.cs
- Common/DTS.Common.Import/Parsers/CSV/Version4CSVSensorParser.cs
- Common/DTS.Common.Import/Parsers/CSV/CSVGroupImport.cs
- Common/DTS.Common.Import/Parsers/CSV/DTSCSVTestSetupParser.cs
- Common/DTS.Common.Import/Parsers/CSV/DTSCSVSensorsParser.cs
generated_at: "2026-04-16T11:47:30.258239+00:00"
model: "zai-org/GLM-5-FP8"
schema_version: 1
sha256: "2bff5b23b8f73415"
---
# CSV Import Module Documentation
## 1. Purpose
This module provides a versioned parser architecture for importing DTS sensor configurations and test setups from CSV files. It transforms raw CSV data into domain objects (`SensorData`, `SensorCalibration`, `TestTemplate`) using a strategy pattern where specific parser versions (e.g., `Version0CSVTestParser`, `Version4CSVSensorParser`) handle different schema evolutions of the CSV format. It orchestrates the creation of groups, channel mappings, and hardware associations required for test setup imports.
## 2. Public Interface
### AbstractCSVParser
Base class for sensor parsers.
* `int Version { get; }` - Abstract property indicating the parser version.
* `void Initialize(ICalibrationImport import, ZeroMethodOptions zmOptions, IImportNotification importNotification, bool importCreateDynamicGroups, bool useISOCodeFilterMapping, bool useZeroForUnfiltered)` - Injects dependencies and configuration flags required for parsing.
* `void ParseVersion(CSVImportTags.Tags field, string val, ParseParameters pp)` - Abstract method to parse a specific field/value pair into the `ParseParameters` state object.
### CSVFile
Utility class for file inspection.
* `CSVFile(string filename)` - Constructor.
* `static bool IsInUse(string filename)` - Attempts to read the file to check for locks; returns `true` if `IOException` occurs.
* `static bool IsCSVFileForTestSetupImport(string filename, CsvImportOptions csvImportOptions)` - Validates if the file starts with the "Version" header.
* `static int GetCsvVersion(string filename)` - Extracts the integer version number from the second line of the CSV.
* `int LineCount { get; }` - Returns the total line count of the file.
### Versioned Test Parsers
Classes implementing `IParseCSVTest`:
* `Version0CSVTestParser`
* `Version5CSVTestParser`
* `Version6CSVTestParser`
**Common Members:**
* `int Version { get; }` - Returns the specific version number (0, 5, or 6).
* `void ParseVersion(CsvReader csvReader, TestSetupImportData tsid)` - Reads the CSV stream and populates `TestSetupImportData`.
### Versioned Sensor Parsers
Classes inheriting `AbstractCSVParser`:
* `Version3CSVSensorParser`: Handles `GroupName`, `GroupType`.
* `Version4CSVSensorParser`: Handles DAS mapping (`DASSerialNumber`, `DASChannelIndex`), UDP streaming (`UDPAddress`, `StreamProfile`), UART settings (`BaudRate`, `Parity`), and user codes.
### CSVGroupImport
Implements `IGroupImport`.
* `ParseParameters ParseParameters { get; set; }`
* `Tuple<TestTemplate, List<IGroup>> CreateGroups(List<SensorData> sensors, Dictionary<string, List<TsetSetupImportSensorInfo>> groupSensorLookup, TestTemplate testTemplate, bool createDynamicGroups, List<IGroup> staticGroups, Action<double> setProgress)` - Constructs group hierarchies and channels within a `TestTemplate`.
* `Dictionary<string, List<TsetSetupImportSensorInfo>> GetGroupSensorLookup(...)` - Builds a lookup dictionary mapping group names to sensor import info.
### DTSCSVTestSetupParser
Main orchestrator for test setup imports.
* `DTSCSVTestSetupParser(IImportNotification importNotification, CsvImportOptions csvImportOptions, TestSetupImportData testSetupImportData, IGroupImport groupImport, bool createDynamicGroups)`
* `void Parse(ref ImportObject importObject)` - Entry point. Validates file, parses test setup, creates groups, assigns calibrations.
* `static TestTemplate CreateTestSetup(TestSetupImportData tsid, DASHardware[] allDAS)` - Factory method to create a `TestTemplate` from parsed data.
* `static void UpdateDASSampleRate(TestTemplate t, TestSetupImportData tsid, DASHardware[] allDAS)` - Updates sample rates on the template.
### DTSCSVSensorsParser
Main orchestrator for sensor imports.
* `DTSCSVSensorsParser(IImportNotification importNotification, User user, CsvImportOptions csvImportOptions, ICalibrationImport calibrationImport, ZeroMethodOptions zeroMethodOptions)`
* `bool ImportCreateDynamicGroups { get; set; }`
* `bool UseISOCodeFilterMapping { get; set; }`
* `bool UseZeroForUnfiltered { get; set; }`
* `void Parse(ref ImportObject importObject)` - Entry point. Reads headers, iterates rows, populates `SensorData` and `SensorCalibration`.
## 3. Invariants
* **Version Header**: A valid CSV file for import must contain a "Version" header in the first row, with the version integer in the second row (enforced by `CSVFile.IsCSVFileForTestSetupImport` and `GetCsvVersion`).
* **Parser Selection**: `DTSCSVTestSetupParser` only invokes parsers for versions 1 through 5 if the file version is exactly 6. Otherwise, it defaults to running only the Version 0 parser.
* **Group Uniqueness**: In `Version3CSVSensorParser`, a sensor serial number cannot appear twice in the `SensorGroupNameLookup` or `SensorGroupTypeLookup`; duplicates trigger an error.
* **Version 6 Tag Filtering**: `Version6CSVTestParser` and `DTSCSVSensorsParser` explicitly check `CSVImportTags.GetVersionForTag` and skip tags that do not match their expected version (e.g., Version 6 parser skips Version 5 tags).
* **Hardware Association**: In `DTSCSVTestSetupParser`, hardware is only added to the `ImportObject` if the CSV version is 6 or greater.
## 4. Dependencies
**Internal Dependencies (Namespaces referenced):**
* `DTS.Common.Classes`, `DTS.Common.Classes.Sensors`: Core domain objects (`SensorData`, `SensorCalibration`).
* `DTS.Common.Enums`, `DTS.Common.Enums.Sensors`: Enumeration types (`RecordingModes`, `BridgeType`, `SensUnits`).
* `DTS.Common.Import.Interfaces`: Interfaces `IParseCSVSensor`, `IParseCSVTest`, `IGroupImport`.
* `DTS.Common.Import.ImportOptions`: `CsvImportOptions`, `ZeroMethodOptions`.
* `DTS.SensorDB`: Database operations and storage types.
* `DataPROWin7.DataModel`: Hardware data models (`DASHardware`).
* `DTS.Common.Storage`: `DbOperations`.
**External Libraries:**
* `CsvHelper`: Used via `CsvReader` for reading CSV records.
**Factory Dependencies (referenced but not defined in source):**
* `CSVTestParserFactory`: Creates the array of test parsers.
* `CSVSensorParserFactory`: Creates the dictionary of sensor parsers.
## 5. Gotchas
* **Silent Exception Swallowing**: `CSVFile.IsInUse` catches all non-IOExceptions and returns `true` (indicating the file is in use/unavailable), which may mask permission errors or corrupt file issues.
* **Hardcoded Version Logic**: `DTSCSVTestSetupParser.ParseTestSetup` contains logic that specifically checks `if (tsid.Version == 6)` to determine whether to iterate through higher-version parsers. Adding a Version 7 would require modifying this specific method.
* **Generic Exception Throwing**: `Version3CSVSensorParser` throws a generic `System.Exception("Parse error")` when a group maps to multiple test objects while dynamic groups are disabled. This is caught in `DTSCSVTestSetupParser.AssignGroupsToTestSetup` and reported generically.
* **Sensitivity Unit Mutation**: `DTSCSVSensorsParser` silently changes `SensitivityUnits` from `mVperVperEU` to `mVperEU` for non-proportional sensors, reporting it only as a non-blocking notification after processing.
* **ISO Channel Prefix**: Sensors with serial numbers starting with `Constants.ISO_CH_ONLY_PREFIX` are excluded from the standard sensor/calibration lookup dictionaries in `DTSCSVSensorsParser`,

182
enriched-partialglm/Common/DTS.Common.Import/Parsers/EQX.md Normal file
View File

@@ -0,0 +1,182 @@
---
source_files:
- Common/DTS.Common.Import/Parsers/EQX/EQXTestSetupParser.cs
- Common/DTS.Common.Import/Parsers/EQX/EQXGroupImport.cs
- Common/DTS.Common.Import/Parsers/EQX/EQXSensorsParser.cs
generated_at: "2026-04-16T11:47:07.484549+00:00"
model: "zai-org/GLM-5-FP8"
schema_version: 1
sha256: "ce57e1c33a0387ff"
---
# EQX Import Parsers Documentation
## 1. Purpose
This module provides parsing functionality for EQX (Equipment Exchange) format files, enabling import of sensor data and test setup configurations into the DTS system. It consists of three main components: `EQXSensorsParser` for parsing sensor definitions and calibrations, `EQXTestSetupParser` for constructing test setups with associated groups and channels, and `EQXGroupImport` for handling group creation and sensor-to-group mapping logic. The module bridges the proprietary EQX XML format with the internal `ImportObject` data model.
---
## 2. Public Interface
### EQXTestSetupParser
**Constructor:**
```csharp
public EQXTestSetupParser(
IImportNotification importNotification,
EqxImportOptions eqxImportOptions,
IGroupImport groupImport,
EquipmentExchange.EQXSensorDatabase eqxSensorDatabase,
bool createDynamicGroups)
```
**Public Methods:**
```csharp
public override void Parse(ref ImportObject importObject)
```
Parses the EQX file specified by `FileName` (inherited from `ParseVariantBase`) and populates the `importObject` with test setup data, groups, and sensors. Validates that the file contains test setup data (checks `GroupNameTestObjectLookup`). Aborts with a critical error if no test setup data is present.
---
### EQXGroupImport
**Implements:** `IGroupImport`
**Public Properties:**
```csharp
public ParseParameters ParseParameters { get; set; }
```
**Public Methods:**
```csharp
public Tuple<TestTemplate, List<IGroup>> CreateGroups(
List<SensorData> sensors,
Dictionary<string, List<TsetSetupImportSensorInfo>> groupSensorLookup,
TestTemplate testTemplate,
bool createDynamicGroups,
List<IGroup> staticGroups,
Action<double> setProgress)
```
Creates groups and populates a `TestTemplate` from sensor data. Returns a tuple containing the configured test template and list of static groups. Returns `null` if `groupSensorLookup` is null or empty. Groups are sorted by sensor count (descending), then by name (ascending) for tie-breaking.
```csharp
public Dictionary<string, List<TsetSetupImportSensorInfo>> GetGroupSensorLookup(
List<SensorData> sensors,
Dictionary<string, string> sensorGroupNameLookup,
Dictionary<string, List<string>> groupNameSensorListLookup)
```
Builds a lookup dictionary mapping group names to sensor information. Mutually exclusive: either `sensorGroupNameLookup` or `groupNameSensorListLookup` should be populated (per FB 30358).
---
### EQXSensorsParser
**Constructor:**
```csharp
public EQXSensorsParser(
IImportNotification importNotification,
User user,
EqxImportOptions eqxImportOptions,
EquipmentExchange.EQXSensorDatabase eqxSensorDatabase)
```
**Constants:**
```csharp
const float MAX_EQX_VERSION_SUPPORT = 1.5F;
```
**Public Properties:**
```csharp
public bool ImportCreateDynamicGroups { get; set; }
public bool EQXUseSerialNumberFieldForSN { get; set; }
public bool UseZeroForUnfiltered { get; set; }
```
**Public Methods:**
```csharp
public override void Parse(ref ImportObject importObject)
```
Parses the EQX file for sensor data. Validates XML structure and EQX format version (must be ≤ 1.5). Populates `importObject` with sensors, calibrations, and sensor model lookups.
---
## 3. Invariants
- **EQX Version Constraint:** `EQXSensorsParser` only supports EQX files with `DataFormatEdition` ≤ `MAX_EQX_VERSION_SUPPORT` (1.5). Files with higher versions trigger an error and abort parsing.
- **Test Setup Presence:** `EQXTestSetupParser.Parse()` requires `GroupNameTestObjectLookup` to be non-null and non-empty. If empty, the import aborts with a critical error (`Import_EQXFileNotForTestSetup`).
- **Null Argument Handling:** Both `Parse()` methods throw `ArgumentNullException` if `importObject` is null. `EQXTestSetupParser.Parse()` returns early (no-op) if `FileName` is null or empty.
- **Calibration Sorting:** In `FixCalibrations()`, calibrations are sorted before the first one is assigned to a sensor. This implies calibrations have a defined sort order (likely by date or version).
- **Group Ordering:** In `EQXGroupImport.CreateGroups()`, groups are sorted by sensor count (descending), then alphabetically by name for equal counts.
- **Mutually Exclusive Lookups:** Per FB 30358, `GetGroupSensorLookup()` expects either `sensorGroupNameLookup` OR `groupNameSensorListLookup` to be populated, not both.
- **Sensor Uniqueness in Groups:** In `CreateGroups()`, a `HashSet<string>` (`sensorSerialHash`) ensures each sensor serial number is only counted once per group for `testGroupChannels`.
---
## 4. Dependencies
### Direct Dependencies (Imports):
**EQXTestSetupParser:**
- `DataPROWin7.DataModel` - `ImportObject`, `ImportError`, `ImportSeverityError`, `TestTemplate`, `SensorData`
- `DTS.Common.Import.ImportOptions` - `EqxImportOptions`
- `DTS.Common.Import.Parsers` - `ParseVariantBase`
- `DTS.Common.Interface.Groups.GroupList` - `IGroupImport`, `IGroup`
- `DTS.Common.SharedResource.Strings` - `StringResources`
- `DTS.Common.Utilities.Logging` - `APILogger`
- `DTS.Common.Utils` - `GroupHelper`
- `DTS.SensorDB` - `EquipmentExchange.EQXSensorDatabase`
- `System.Xml.Linq` - XML parsing
**EQXGroupImport:**
- `DataPROWin7.DataModel` - `TestTemplate`, `SensorData`, `TsetSetupImportSensorInfo`
- `DTS.Common.Classes.Groups` - Group-related classes
- `DTS.Common.Classes.Sensors` - `GroupChannel`
- `DTS.Common.Interface.Groups.GroupList` - `IGroupImport`, `IGroup`
- `DTS.Common.Interface.Channels` - `IGroupChannel`
- `DTS.Common.Storage` - `DbOperations`
- `DTS.SensorDB` - Sensor database types
**EQXSensorsParser:**
- `DTS.Common.Enums` - `BridgeType`, `SquibMeasurementType`, `ExcitationVoltageOptions`
- `DTS.Common.Import.ImportOptions` - `EqxImportOptions`
- `DTS.Common.Import.Parsers` - `ParseVariantBase`
- `DTS.Common.Storage` - `DbOperations`
- `DTS.Common.Classes.Sensors` - `FactorySensorModel`, `SensorsCollection`
- `DTS.Common.Interface.Sensors` - `ISensorData`
- `DTS.Common.SharedResource.Strings` - `StringResources`
- `DTS.Slice.Users` - `User`
- `DTS.SensorDB` - `EquipmentExchange.EQXSensorDatabase`
### Consumers:
- The module is designed to be consumed by the DTS import infrastructure via `ParseVariantBase` inheritance pattern.
- `EQXGroupImport` implements `IGroupImport`, suggesting it's injected into consumers that depend on that interface.
---
## 5. Gotchas
1. **Calibration Reset Bug (Case 44105):** `NormalizeSensorIds()` in `GroupHelper` can reset sensor calibrations. `EQXTestSetupParser.AssignGroupsToTestSetup()` calls `FixCalibrations()` after `NormalizeSensorIds()` to restore them. This is a known workaround documented in the source.
2. **Duplicate Sensor Addition:** In `EQXSensorsParser.ParseSensor()`, sensors are added to `importObject` via both `AddSensorLookup()` and `AddSensor()`. The comment "this is for compatability for import test setups" suggests legacy behavior being maintained.
3. **EID Preservation (Case 18467):** If an EQX file has a null `IDModuleString`, the parser attempts to preserve the existing EID from the sensor database to avoid wiping it out.
4. **Excitation Errors Not Reported:** `GetExcitationErrors()` collects excitation validation errors, but the method includes a `//TODO Report the errors` comment, indicating these errors may not be surfaced to users.
5. **Silent XML Parsing Failure:** `ParseSetupName()` catches all exceptions and returns `string.Empty` without logging, making setup name extraction failures invisible.
6. **Static Group Mutation:** `EQXGroupImport.CreateGroups()` modifies the `staticGroups` list parameter in-place (calls `staticGroups.Add(newGroup)`) rather than returning a new collection.
7. **Progress Callback Frequency:** In `EQXSensorsParser.ParseSensor()`, progress updates only occur every 10 iterations (`if (0 == currentDone % 10)`), which may cause UI staleness for small imports.
8. **Zero Method Hardcoded Index:** In `CreateGroups()`, zero method properties are accessed via `Methods[0]` without bounds checking, assuming at least one method exists in `ZeroMethods.Methods`.