--- source_files: - Common/DTS.Common.Import/Parsers/CSV/AbstractCSVParser.cs - Common/DTS.Common.Import/Parsers/CSV/Version0CSVTestParser.cs - Common/DTS.Common.Import/Parsers/CSV/CSVFile.cs - Common/DTS.Common.Import/Parsers/CSV/Version6CSVTestParser.cs - Common/DTS.Common.Import/Parsers/CSV/Version3CSVSensorParser.cs - Common/DTS.Common.Import/Parsers/CSV/Version5CSVTestParser.cs - Common/DTS.Common.Import/Parsers/CSV/Version4CSVSensorParser.cs - Common/DTS.Common.Import/Parsers/CSV/CSVGroupImport.cs - Common/DTS.Common.Import/Parsers/CSV/DTSCSVTestSetupParser.cs - Common/DTS.Common.Import/Parsers/CSV/DTSCSVSensorsParser.cs generated_at: "2026-04-16T02:08:24.981291+00:00" model: "Qwen/Qwen3-Coder-Next-FP8" schema_version: 1 sha256: "2bff5b23b8f73415" --- # CSV Import Module Documentation ## 1. Purpose This module provides a versioned CSV parsing infrastructure for importing sensor and test setup data into the DTS system. It supports two distinct import workflows: **sensor data import** (handled by `DTSCSVSensorsParser`) and **test setup data import** (handled by `DTSCSVTestSetupParser`). The module uses an abstract base class (`AbstractCSVParser`) and concrete version-specific parsers to handle evolving CSV formats across versions 0 through 6. It decouples parsing logic from business logic, delegates group creation to `CSVGroupImport`, and integrates with calibration, sensor, and hardware data models to construct `TestTemplate` and `ImportObject` instances for downstream processing. ## 2. Public Interface ### `AbstractCSVParser` (abstract base class) - **`int Version { get; }`** Returns the CSV format version supported by this parser (e.g., 3, 4). - **`void Initialize(ICalibrationImport import, ZeroMethodOptions zmOptions, IImportNotification importNotification, bool importCreateDynamicGroups, bool useISOCodeFilterMapping, bool useZeroForUnfiltered)`** Injects dependencies and configuration flags required for parsing. Must be called before `ParseVersion`. - **`abstract void ParseVersion(CSVImportTags.Tags field, string val, ParseParameters pp)`** Parses a single field (`field`) from a sensor CSV row, using value `val`. Errors are appended to `pp.Errors`. Behavior is version-specific. ### `Version0CSVTestParser` - **`int Version => 0`** - **`void ParseVersion(CsvReader csvReader, TestSetupImportData tsid)`** Parses test setup metadata (e.g., `PostTriggerSec`, `SampleRate`, `RecordingMode`) from a CSV with a simple header-value pair format. Reads two rows: header names and values. ### `Version3CSVSensorParser` - **`int Version => 3`** - **`void ParseVersion(CSVImportTags.Tags field, string sVal, ParseParameters pp)`** Handles `GroupName` and `GroupType` fields. Enforces non-empty values, uniqueness of sensor serial numbers in group lookups, and consistency of `TestObject` per group (throws `Exception("Parse error")` on conflict if `ImportCreateDynamicGroups` is false). ### `Version4CSVSensorParser` - **`int Version => 4`** - **`void ParseVersion(CSVImportTags.Tags field, string sVal, ParseParameters pp)`** Parses DAS-related (`DASSerialNumber`, `DASChannelIndex`), streaming (`StreamProfile`, `UDPAddress`, `TimeChannelId`, `DataChannelId`, `TmNSConfig`, `IRIGTimeDataPacketIntervalMS`, `TMATSIntervalMS`), UART (`BaudRate`, `DataBits`, `StopBits`, `Parity`, `DataFormat`), and user-defined fields (`TestUserCode`, `TestUserChannelName`, `TestIsoCode`, `TestIsoChannelName`). Updates `pp.SensorData` and lookup dictionaries. ### `Version5CSVTestParser` - **`int Version => 5`** - **`void ParseVersion(CsvReader csvReader, TestSetupImportData tsid)`** Parses clock synchronization settings (e.g., `ClockMasterInputType`, `ClockMasterOutputType`, `ManageClocksOutsideDPMaster`) from a header-value row format. ### `Version6CSVTestParser` - **`int Version => 6`** - **`void ParseVersion(CsvReader csvReader, TestSetupImportData tsid)`** Parses per-DAS hardware configuration (e.g., `DASSerial`, `DASSampleRate`, `PTPDomainId`, `ClockMaster`) in a table format. Reads header row, then iterates rows until an empty line or non-matching field is encountered. ### `CSVFile` - **`static bool IsInUse(string filename)`** Returns `true` if the file is locked (IOException on `File.ReadAllLines`), else `false`. - **`static bool IsCSVFileForTestSetupImport(string filename, CsvImportOptions csvImportOptions)`** Returns `true` if the first field of the first row is `"Version"`. - **`static int GetCsvVersion(string filename)`** Returns the integer value in the first column of the second row if the first row starts with `"Version"`, else `0`. - **`int LineCount { get; }`** Returns the number of lines in the file (0 on error or null filename). ### `CSVGroupImport` - **`ParseParameters ParseParameters { get; set; }`** - **`Tuple> CreateGroups(List sensors, Dictionary> groupSensorLookup, TestTemplate testTemplate, bool createDynamicGroups, List staticGroups, Action setProgress)`** Creates `IGroup` instances from `groupSensorLookup`, assigns sensor/channel properties (ISO/user codes, DAS mapping, calibration defaults), and adds groups to `testTemplate`. Updates `setProgress`. - **`Dictionary> GetGroupSensorLookup(List sensors, Dictionary sensorGroupNameLookup, Dictionary> groupNameSensorListLookup)`** Builds a mapping from group name to list of `TsetSetupImportSensorInfo` (containing sensor/DAS parameters) using either `sensorGroupNameLookup` or `groupNameSensorListLookup`. ### `DTSCSVTestSetupParser` - **`void Parse(ref ImportObject importObject)`** Orchestrates test setup import: validates CSV format, parses test setup data via `ParseTestSetup`, creates `TestTemplate`, assigns hardware, clock sync, and tags, then delegates group creation to `AssignGroupsToTestSetup`. Adds result to `importObject`. ### `DTSCSVSensorsParser` - **`void Parse(ref ImportObject importObject)`** Orchestrates sensor import: reads CSV, populates `SensorData` and `SensorCalibration` objects via `PopulateSensor`, handles sensitivity unit corrections, and adds sensors/calibrations to `importObject`. Reports errors and warnings. ## 3. Invariants - **CSV Format Validation**: - A CSV is recognized as a test setup file if its first field is `"Version"`. - Sensor CSVs must have recognizable column headers (else `ImportSensorsPreviewControl_NoColumnsInTDCCSV` error). - **Version Consistency**: - `Version0CSVTestParser` and `Version5CSVTestParser` expect header-value pairs. - `Version6CSVTestParser` expects a table format with header row followed by data rows until an empty line. - `DTSCSVTestSetupParser.ParseTestSetup` uses `CSVFile.GetCsvVersion` to determine which parsers to invoke; version 6 triggers all parsers (0 + 5 + 6). - **Sensor Grouping**: - `Version3CSVSensorParser` enforces that a group name maps to a single `SensorTestObject` unless `ImportCreateDynamicGroups` is true. - Duplicate sensor serial numbers in `sensorGroupNameLookup` or `sensorGroupTypeLookup` are rejected with errors. - **Calibration & Defaults**: - `DTSCSVSensorsParser` applies user-specific defaults for squib/digital output settings. - Sensitivity units are corrected from `mVperVperEU` to `mVperEU` for non-proportional calibrations. - Zero method is initialized from `pp.SensorCal.ZeroMethods.Methods[0]` during `PostParse`. - **Hardware Mapping**: - DAS hardware is added to `TestTemplate` only if version ≥ 6 (per `DTSCSVTestSetupParser` comment FB 43815). - DAS sample rates are set in both `t.SetSampleRateForHardware` and `t.DASSampleRateList`. ## 4. Dependencies ### Imports/References: - **Core Types**: `DTS.Common.Classes`, `DTS.Common.Enums`, `DTS.Common.Import.Interfaces`, `CsvHelper`, `System.IO`, `System.Collections.Generic`. - **Hardware/Storage**: `DataPROWin7.DataModel`, `DTS.SensorDB`, `DTS.Common.Storage`. - **Import Infrastructure**: `ParseVariantBase`, `ParseParameters`, `ImportObject`, `TestSetupImportData`, `SensorData`, `SensorCalibration`, `IGroup`, `IGroupImport`, `ICalibrationImport`, `IImportNotification`. - **CSV Tags**: `CSVImportTags` (static class providing `GetTagForString`, `GetVersionForTag`). - **Factories**: `CSVTestParserFactory`, `CSVSensorParserFactory`. ### Used By: - `DTSCSVTestSetupParser` is invoked by the test setup import pipeline. - `DTSCSVSensorsParser` is invoked by the sensor import pipeline. - `CSVGroupImport` is used by `DTSCSVTestSetupParser` to create groups. ## 5. Gotchas - **Version 6 CSV Parsing Quirk**: In `DTSCSVSensorsParser.ParseSensor`, if version is 6, the parser skips rows until it finds a tag with version ≤ 4 (to avoid parsing test setup-specific rows as sensor data). This assumes sensor and test setup data are in the same file but separated by an empty line. - **Exception Thrown in Group Parsing**: `Version3CSVSensorParser.ParseVersion` throws `new Exception("Parse error")` on group/test object conflict (if `ImportCreateDynamicGroups` is false). This is caught by `DTSCSVSensorsParser` and reported as a sensor error, but the exception itself is non-descriptive. - **Sensitivity Unit Correction**: `DTSCSVSensorsParser` silently changes `SensitivityUnits` from `mVperVperEU` to `mVperEU` for non-proportional calibrations. This is logged as a warning but may cause unexpected calibration behavior. - **DAS Hardware Lookup**: `CSVGroupImport.CreateGroups` uses `DbOperations.GetChannelSettingDefaults()` and `GroupHelper.GetDAS` to map DAS serial numbers to IDs. If DAS is not in `DASHardwareList.GetAllHardware()`, channel DAS ID remains unset. - **Zero Method Initialization**: `PostParse` in `DTSCSVSensorsParser` initializes `pp.ZeroType`, `pp.ZeroStart`, `pp.ZeroEnd` from `pp.SensorCal.ZeroMethods.Methods[0]`. If `ZeroMethods` is uninitialized or empty, this may cause `IndexOutOfRangeException`. - **No Validation of Sensor Serial Number Format**: Sensors with serial numbers starting with `Constants.ISO_CH_ONLY_PREFIX` bypass some validation (e.g., sensitivity/capacity checks), but the prefix is not defined in the provided source. - **Error Reporting Inconsistency**: `DTSCSVSensorsParser` aggregates errors per line and reports them as a single error string, while `DTSCSVTestSetupParser` reports errors via `IImportNotification.ReportErrors`. Error severity and continuation behavior differ between parsers. - **File Lock Detection Limitation**: `CSVFile.IsInUse` uses `File.ReadAllLines`, which may not reliably detect all lock states (e.g., file opened for append-only).