noisedestroyers/DP44
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
bc3ac1d4c9bfb56c00ec892a5a321bb0e5dcc55d
DP44/docs/ai/Common/DTS.Common.Import/Parsers/CSV.md
noisedestroyers bc3ac1d4c9 init
2026-04-17 14:55:32 -04:00

182 lines
13 KiB
Markdown
Raw Blame History

---
source_files:
- Common/DTS.Common.Import/Parsers/CSV/AbstractCSVParser.cs
- Common/DTS.Common.Import/Parsers/CSV/Version0CSVTestParser.cs
- Common/DTS.Common.Import/Parsers/CSV/CSVFile.cs
- Common/DTS.Common.Import/Parsers/CSV/Version6CSVTestParser.cs
- Common/DTS.Common.Import/Parsers/CSV/Version3CSVSensorParser.cs
- Common/DTS.Common.Import/Parsers/CSV/Version5CSVTestParser.cs
- Common/DTS.Common.Import/Parsers/CSV/Version4CSVSensorParser.cs
- Common/DTS.Common.Import/Parsers/CSV/CSVGroupImport.cs
- Common/DTS.Common.Import/Parsers/CSV/DTSCSVTestSetupParser.cs
- Common/DTS.Common.Import/Parsers/CSV/DTSCSVSensorsParser.cs
generated_at: "2026-04-17T15:31:35.696653+00:00"
model: "zai-org/GLM-5-FP8"
schema_version: 1
sha256: "82eec24579c17b93"
---
# CSV Import Parsers Documentation
## 1. Purpose
This module provides CSV parsing functionality for importing DTS sensor configurations and test setups. It implements a versioned parser architecture where different CSV format versions (0-6) are handled by specialized parser classes. The module reads CSV files containing sensor metadata, calibration data, group assignments, and test setup configurations, transforming them into domain objects for the DTS data acquisition system. The architecture separates sensor parsing from test setup parsing, with both feeding into a group import system that creates the final test template structure.
---
## 2. Public Interface
### AbstractCSVParser (Abstract Base Class)
| Member | Signature | Description |
|--------|-----------|-------------|
| Version | `public abstract int Version { get; }` | Returns the parser version number this class handles |
| Initialize | `public void Initialize(ICalibrationImport import, ZeroMethodOptions zmOptions, IImportNotification importNotification, bool importCreateDynamicGroups, bool useISOCodeFilterMapping, bool useZeroForUnfiltered)` | Initializes the parser with dependencies and configuration flags |
| ParseVersion | `public abstract void ParseVersion(CSVImportTags.Tags field, string val, ParseParameters pp)` | Parses a specific field value for this parser's version |
### Version0CSVTestParser
| Member | Signature | Description |
|--------|-----------|-------------|
| Version | `public int Version => 0` | Returns version 0 |
| ParseVersion | `public void ParseVersion(CsvReader csvReader, TestSetupImportData tsid)` | Parses version 0 test setup data including PostTriggerSec, PreTriggerSec, RecordingMode, SampleRate, Description, Name, Version, and Tags |
### Version5CSVTestParser
| Member | Signature | Description |
|--------|-----------|-------------|
| Version | `public int Version => 5` | Returns version 5 |
| ParseVersion | `public void ParseVersion(CsvReader csvReader, TestSetupImportData tsid)` | Parses version 5 test setup data including clock configuration (ClockMasterInputType, ClockMasterOutputType, ClockSlaveInputType, ClockSlaveOutputType, ManageClocksOutsideDPMaster, ManageClocksOutsideDPSlave) |
### Version6CSVTestParser
| Member | Signature | Description |
|--------|-----------|-------------|
| Version | `public int Version => 6` | Returns version 6 |
| ParseVersion | `public void ParseVersion(CsvReader csvReader, TestSetupImportData tsid)` | Parses version 6 test setup data including DAS hardware info (DASSerial, DASSampleRate, PTPDomainId, ClockMaster) with multi-row body parsing |
### Version3CSVSensorParser
| Member | Signature | Description |
|--------|-----------|-------------|
| Version | `public override int Version => 3` | Returns version 3 |
| ParseVersion | `public override void ParseVersion(CSVImportTags.Tags field, string sVal, ParseParameters pp)` | Parses GroupName and GroupType fields with duplicate detection and test object validation |
### Version4CSVSensorParser
| Member | Signature | Description |
|--------|-----------|-------------|
| Version | `public override int Version => 4` | Returns version 4 |
| ParseVersion | `public override void ParseVersion(CSVImportTags.Tags field, string sVal, ParseParameters pp)` | Parses DAS assignment (DASSerialNumber, DASChannelIndex), UDP streaming (StreamProfile, UDPAddress, TimeChannelId, DataChannelId, TmNSConfig, IRIGTimeDataPacketIntervalMS, TMATSIntervalMS), UART configuration (BaudRate, DataBits, StopBits, Parity, DataFormat), and user codes (TestUserCode, TestUserChannelName, TestIsoCode, TestIsoChannelName) |
### CSVFile
| Member | Signature | Description |
|--------|-----------|-------------|
| Constructor | `public CSVFile(string filename)` | Creates instance with specified filename |
| IsInUse | `public static bool IsInUse(string filename)` | Returns true if file is locked/in use by another process |
| IsCSVFileForTestSetupImport | `public static bool IsCSVFileForTestSetupImport(string filename, CsvImportOptions csvImportOptions)` | Validates if CSV file is formatted for test setup import (checks for "Version" header) |
| GetCsvVersion | `public static int GetCsvVersion(string filename)` | Extracts and returns the version number from CSV file, returns 0 on error |
| LineCount | `public int LineCount { get; }` | Returns the number of lines in the file |
### CSVGroupImport
| Member | Signature | Description |
|--------|-----------|-------------|
| ParseParameters | `public ParseParameters ParseParameters { get; set; }` | Gets or sets the parse parameters for import |
| CreateGroups | `public Tuple<TestTemplate, List<IGroup>> CreateGroups(List<SensorData> sensors, Dictionary<string, List<TsetSetupImportSensorInfo>> groupSensorLookup, TestTemplate testTemplate, bool createDynamicGroups, List<IGroup> staticGroups, Action<double> setProgress)` | Creates groups and channels from sensor data, populating the test template |
| GetGroupSensorLookup | `public Dictionary<string, List<TsetSetupImportSensorInfo>> GetGroupSensorLookup(List<SensorData> sensors, Dictionary<string, string> sensorGroupNameLookup, Dictionary<string, List<string>> groupNameSensorListLookup)` | Builds lookup dictionary mapping group names to sensor info lists |
### DTSCSVTestSetupParser
| Member | Signature | Description |
|--------|-----------|-------------|
| Constructor | `public DTSCSVTestSetupParser(IImportNotification importNotification, CsvImportOptions csvImportOptions, TestSetupImportData testSetupImportData, IGroupImport groupImport, bool createDynamicGroups)` | Initializes parser with dependencies |
| CreateTestSetup | `public static TestTemplate CreateTestSetup(TestSetupImportData tsid, DASHardware[] allDAS)` | Creates a TestTemplate from parsed import data |
| UpdateDASSampleRate | `public static void UpdateDASSampleRate(TestTemplate t, TestSetupImportData tsid, DASHardware[] allDAS)` | Updates sample rates for DAS hardware in the test template |
| Parse | `public override void Parse(ref ImportObject importObject)` | Main entry point - parses CSV file and populates ImportObject with test setup data |
### DTSCSVSensorsParser
| Member | Signature | Description |
|--------|-----------|-------------|
| Constructor | `public DTSCSVSensorsParser(IImportNotification importNotification, User user, CsvImportOptions csvImportOptions, ICalibrationImport calibrationImport, ZeroMethodOptions zeroMethodOptions)` | Initializes parser with dependencies |
| ImportCreateDynamicGroups | `public bool ImportCreateDynamicGroups { get; set; }` | Flag for dynamic group creation during import |
| UseISOCodeFilterMapping | `public bool UseISOCodeFilterMapping { get; set; }` | Flag for ISO code filter mapping behavior |
| UseZeroForUnfiltered | `public bool UseZeroForUnfiltered { get; set; }` | Flag for using zero values for unfiltered data |
| Parse | `public override void Parse(ref ImportObject importObject)` | Main entry point - parses CSV file and populates ImportObject with sensor and calibration data |
---
## 3. Invariants
1. **Version Header Requirement**: CSV files must have "Version" as the first token in the first row to be recognized as valid for import.
2. **Version-Based Parser Selection**: The `CSVImportTags.GetVersionForTag()` determines which parser handles each field. Parsers only process fields matching their version.
3. **Version 6 Multi-Parser Execution**: When `tsid.Version == 6`, all parsers from index 1 through the array length are invoked sequentially (see `DTSCSVTestSetupParser.ParseTestSetup`).
4. **Duplicate Serial Number Detection**: `Version3CSVSensorParser` enforces unique sensor serial numbers within `SensorGroupNameLookup` and `SensorGroupTypeLookup` dictionaries.
5. **Non-Empty Group Names**: Group names must be non-empty and non-whitespace; empty values generate errors.
6. **Test Setup Name Uniqueness**: Import skips test setups with names that already exist in `importObject.TestSetups()`.
7. **Calibration Assignment Order**: Calibrations are sorted and the most recent (last element after sort) is assigned to each sensor.
8. **Analog Sensor Validation**: Analog sensors require non-zero sensitivity and capacity >= 1; violations generate errors.
9. **Excitation Validation**: Full/Half/Quarter bridge sensors require defined excitation voltage (not `Undefined`).
---
## 4. Dependencies
### External Dependencies (Imports)
| Namespace | Purpose |
|-----------|---------|
| `CsvHelper` | CSV file reading via `CsvReader` |
| `DTS.Common.Classes.Sensors` | `SensorData`, `SensorCalibration`, `ParseParameters` |
| `DTS.Common.Enums.Sensors` | Sensor enumerations including `BridgeType`, `SensUnits`, `ExcitationVoltageOptions` |
| `DTS.Common.Import.ImportOptions` | `CsvImportOptions`, `ZeroMethodOptions` |
| `DTS.Common.Import.Interfaces` | `IParseCSVSensor`, `IParseCSVTest`, `ICalibrationImport`, `IImportNotification`, `IGroupImport` |
| `DTS.Common.Import.Factories` | `CSVTestParserFactory`, `CSVSensorParserFactory` |
| `DTS.Common.Import.Parsers` | `ParseVariantBase` base class |
| `DTS.Common.Storage` | `DbOperations` for database access |
| `DTS.SensorDB` | Database operations |
| `DataPROWin7.DataModel.Classes.Hardware` | `DASHardwareList` for hardware enumeration |
| `System.IO.Ports` | `StopBits`, `Parity` enumerations for UART configuration |
| `DTS.Slice.Users` | `User` class for current user context |
| `DTS.Common.Utilities.Logging` | `APILogger` for error logging |
### Consumers (Inferred)
The module implements interfaces (`IParseCSVSensor`, `IParseCSVTest`, `IGroupImport`) and extends `ParseVariantBase`, suggesting consumption via:
- Factory patterns (`CSVTestParserFactory`, `CSVSensorParserFactory`)
- Dependency injection through interface types
- The `ImportObject` pattern which aggregates parsed results
---
## 5. Gotchas
1. **Version 6 Special Execution Path**: Unlike other versions, version 6 triggers iteration through all parsers starting at index 1:
```csharp
if (tsid.Version == 6)
{
for (var i = 1; i < parsers.Length; i++)
{
parsers[i].ParseVersion(parser, tsid);
}
}
```
This means version 6 files are processed by multiple parsers, not just `Version6CSVTestParser`.
2. **Silent Sensitivity Unit Correction**: Non-proportional sensors with `SensUnits.mVperVperEU` are silently corrected to `SensUnits.mVperEU`. Users are notified via `_importNotification.ReportErrors` but parsing continues.
3. **ISO Channel-Only Prefix Bypass**: Sensors with serial numbers starting with `Constants.ISO_CH_ONLY_PREFIX` are excluded from `AddSensorLookup` and `AddCalibrationLookup` calls, but still added via `AddSensor` and `AddCalibration`.
4. **Exception Swallowing in CSVFile**:
- `IsInUse()` suppresses all non-IOException exceptions
- `IsCSVFileForTestSetupImport()` and `GetCsvVersion()` catch all exceptions and return `false`/`0`
5. **Generic Exception in Version3CSVSensorParser**: Throws `new Exception("Parse error")` when a group maps to multiple test objects and `ImportCreateDynamicGroups` is false. This breaks the pattern of adding errors to `pp.Errors`.
6. **Dual Sample Rate Storage**: The comment in `UpdateDASSampleRate` explicitly notes uncertainty about why sample rates are stored in both `t.SetSampleRateForHardware()` and `t.DASSampleRateList[]`.
7. **Historical Bug References**: Comments reference "FB" tracking numbers (e.g., "FB 40598", "FB 36879", "FB 43815") indicating accumulated fixes. These may indicate areas with historical issues.
8. **Range Assignment Complexity**: The range assignment logic in `CSVGroupImport.CreateGroups` has specific conditions involving `ImportContainedSensorRanges`, `RangeHigh`, `RangeLow`, `RangeMedium`, and `Capacity` that may produce unexpected results if not fully understood.
9. **Version 6 Header Search Loop**: `Version6CSVTestParser` and `Version5CSVTestParser` both skip rows until finding a version-appropriate tag, but use different version checks (6 vs 5). This could cause confusion if tags overlap.
Reference in New Issue View Git Blame Copy Permalink