DP44/enriched-qwen3-coder-next/Common/DTS.Common.Import/Parsers/CSV.md

source_files, generated_at, model, schema_version, sha256

source_files	generated_at	model	schema_version	sha256
Common/DTS.Common.Import/Parsers/CSV/AbstractCSVParser.cs Common/DTS.Common.Import/Parsers/CSV/Version0CSVTestParser.cs Common/DTS.Common.Import/Parsers/CSV/CSVFile.cs Common/DTS.Common.Import/Parsers/CSV/Version6CSVTestParser.cs Common/DTS.Common.Import/Parsers/CSV/Version3CSVSensorParser.cs Common/DTS.Common.Import/Parsers/CSV/Version5CSVTestParser.cs Common/DTS.Common.Import/Parsers/CSV/Version4CSVSensorParser.cs Common/DTS.Common.Import/Parsers/CSV/CSVGroupImport.cs Common/DTS.Common.Import/Parsers/CSV/DTSCSVTestSetupParser.cs Common/DTS.Common.Import/Parsers/CSV/DTSCSVSensorsParser.cs	2026-04-16T02:08:24.981291+00:00	Qwen/Qwen3-Coder-Next-FP8	1	2bff5b23b8f73415

CSV Import Module Documentation

1. Purpose

This module provides a versioned CSV parsing infrastructure for importing sensor and test setup data into the DTS system. It supports two distinct import workflows: sensor data import (handled by DTSCSVSensorsParser) and test setup data import (handled by DTSCSVTestSetupParser). The module uses an abstract base class (AbstractCSVParser) and concrete version-specific parsers to handle evolving CSV formats across versions 0 through 6. It decouples parsing logic from business logic, delegates group creation to CSVGroupImport, and integrates with calibration, sensor, and hardware data models to construct TestTemplate and ImportObject instances for downstream processing.

2. Public Interface

AbstractCSVParser (abstract base class)

• int Version { get; }
  Returns the CSV format version supported by this parser (e.g., 3, 4).
• void Initialize(ICalibrationImport import, ZeroMethodOptions zmOptions, IImportNotification importNotification, bool importCreateDynamicGroups, bool useISOCodeFilterMapping, bool useZeroForUnfiltered)
  Injects dependencies and configuration flags required for parsing. Must be called before ParseVersion.
• abstract void ParseVersion(CSVImportTags.Tags field, string val, ParseParameters pp)
  Parses a single field (field) from a sensor CSV row, using value val. Errors are appended to pp.Errors. Behavior is version-specific.

Version0CSVTestParser

• int Version => 0
• void ParseVersion(CsvReader csvReader, TestSetupImportData tsid)
  Parses test setup metadata (e.g., PostTriggerSec, SampleRate, RecordingMode) from a CSV with a simple header-value pair format. Reads two rows: header names and values.

Version3CSVSensorParser

• int Version => 3
• void ParseVersion(CSVImportTags.Tags field, string sVal, ParseParameters pp)
  Handles GroupName and GroupType fields. Enforces non-empty values, uniqueness of sensor serial numbers in group lookups, and consistency of TestObject per group (throws Exception("Parse error") on conflict if ImportCreateDynamicGroups is false).

Version4CSVSensorParser

• int Version => 4
• void ParseVersion(CSVImportTags.Tags field, string sVal, ParseParameters pp)
  Parses DAS-related (DASSerialNumber, DASChannelIndex), streaming (StreamProfile, UDPAddress, TimeChannelId, DataChannelId, TmNSConfig, IRIGTimeDataPacketIntervalMS, TMATSIntervalMS), UART (BaudRate, DataBits, StopBits, Parity, DataFormat), and user-defined fields (TestUserCode, TestUserChannelName, TestIsoCode, TestIsoChannelName). Updates pp.SensorData and lookup dictionaries.

Version5CSVTestParser

• int Version => 5
• void ParseVersion(CsvReader csvReader, TestSetupImportData tsid)
  Parses clock synchronization settings (e.g., ClockMasterInputType, ClockMasterOutputType, ManageClocksOutsideDPMaster) from a header-value row format.

Version6CSVTestParser

• int Version => 6
• void ParseVersion(CsvReader csvReader, TestSetupImportData tsid)
  Parses per-DAS hardware configuration (e.g., DASSerial, DASSampleRate, PTPDomainId, ClockMaster) in a table format. Reads header row, then iterates rows until an empty line or non-matching field is encountered.

CSVFile

• static bool IsInUse(string filename)
  Returns true if the file is locked (IOException on File.ReadAllLines), else false.
• static bool IsCSVFileForTestSetupImport(string filename, CsvImportOptions csvImportOptions)
  Returns true if the first field of the first row is "Version".
• static int GetCsvVersion(string filename)
  Returns the integer value in the first column of the second row if the first row starts with "Version", else 0.
• int LineCount { get; }
  Returns the number of lines in the file (0 on error or null filename).

CSVGroupImport

• ParseParameters ParseParameters { get; set; }
• Tuple<TestTemplate, List<IGroup>> CreateGroups(List<SensorData> sensors, Dictionary<string, List<TsetSetupImportSensorInfo>> groupSensorLookup, TestTemplate testTemplate, bool createDynamicGroups, List<IGroup> staticGroups, Action<double> setProgress)
  Creates IGroup instances from groupSensorLookup, assigns sensor/channel properties (ISO/user codes, DAS mapping, calibration defaults), and adds groups to testTemplate. Updates setProgress.
• Dictionary<string, List<TsetSetupImportSensorInfo>> GetGroupSensorLookup(List<SensorData> sensors, Dictionary<string, string> sensorGroupNameLookup, Dictionary<string, List<string>> groupNameSensorListLookup)
  Builds a mapping from group name to list of TsetSetupImportSensorInfo (containing sensor/DAS parameters) using either sensorGroupNameLookup or groupNameSensorListLookup.

DTSCSVTestSetupParser

• void Parse(ref ImportObject importObject)
  Orchestrates test setup import: validates CSV format, parses test setup data via ParseTestSetup, creates TestTemplate, assigns hardware, clock sync, and tags, then delegates group creation to AssignGroupsToTestSetup. Adds result to importObject.

DTSCSVSensorsParser

• void Parse(ref ImportObject importObject)
  Orchestrates sensor import: reads CSV, populates SensorData and SensorCalibration objects via PopulateSensor, handles sensitivity unit corrections, and adds sensors/calibrations to importObject. Reports errors and warnings.

3. Invariants

• CSV Format Validation:
  • A CSV is recognized as a test setup file if its first field is "Version".
  • Sensor CSVs must have recognizable column headers (else ImportSensorsPreviewControl_NoColumnsInTDCCSV error).
• Version Consistency:
  • Version0CSVTestParser and Version5CSVTestParser expect header-value pairs.
  • Version6CSVTestParser expects a table format with header row followed by data rows until an empty line.
  • DTSCSVTestSetupParser.ParseTestSetup uses CSVFile.GetCsvVersion to determine which parsers to invoke; version 6 triggers all parsers (0 + 5 + 6).
• Sensor Grouping:
  • Version3CSVSensorParser enforces that a group name maps to a single SensorTestObject unless ImportCreateDynamicGroups is true.
  • Duplicate sensor serial numbers in sensorGroupNameLookup or sensorGroupTypeLookup are rejected with errors.
• Calibration & Defaults:
  • DTSCSVSensorsParser applies user-specific defaults for squib/digital output settings.
  • Sensitivity units are corrected from mVperVperEU to mVperEU for non-proportional calibrations.
  • Zero method is initialized from pp.SensorCal.ZeroMethods.Methods[0] during PostParse.
• Hardware Mapping:
  • DAS hardware is added to TestTemplate only if version ≥ 6 (per DTSCSVTestSetupParser comment FB 43815).
  • DAS sample rates are set in both t.SetSampleRateForHardware and t.DASSampleRateList.

4. Dependencies

Imports/References:

• Core Types: DTS.Common.Classes, DTS.Common.Enums, DTS.Common.Import.Interfaces, CsvHelper, System.IO, System.Collections.Generic.
• Hardware/Storage: DataPROWin7.DataModel, DTS.SensorDB, DTS.Common.Storage.
• Import Infrastructure: ParseVariantBase, ParseParameters, ImportObject, TestSetupImportData, SensorData, SensorCalibration, IGroup, IGroupImport, ICalibrationImport, IImportNotification.
• CSV Tags: CSVImportTags (static class providing GetTagForString, GetVersionForTag).
• Factories: CSVTestParserFactory, CSVSensorParserFactory.

Used By:

• DTSCSVTestSetupParser is invoked by the test setup import pipeline.
• DTSCSVSensorsParser is invoked by the sensor import pipeline.
• CSVGroupImport is used by DTSCSVTestSetupParser to create groups.

5. Gotchas

• Version 6 CSV Parsing Quirk:
  In DTSCSVSensorsParser.ParseSensor, if version is 6, the parser skips rows until it finds a tag with version ≤ 4 (to avoid parsing test setup-specific rows as sensor data). This assumes sensor and test setup data are in the same file but separated by an empty line.

• Exception Thrown in Group Parsing:
  Version3CSVSensorParser.ParseVersion throws new Exception("Parse error") on group/test object conflict (if ImportCreateDynamicGroups is false). This is caught by DTSCSVSensorsParser and reported as a sensor error, but the exception itself is non-descriptive.

• Sensitivity Unit Correction:
  DTSCSVSensorsParser silently changes SensitivityUnits from mVperVperEU to mVperEU for non-proportional calibrations. This is logged as a warning but may cause unexpected calibration behavior.

• DAS Hardware Lookup:
  CSVGroupImport.CreateGroups uses DbOperations.GetChannelSettingDefaults() and GroupHelper.GetDAS to map DAS serial numbers to IDs. If DAS is not in DASHardwareList.GetAllHardware(), channel DAS ID remains unset.

• Zero Method Initialization:
  PostParse in DTSCSVSensorsParser initializes pp.ZeroType, pp.ZeroStart, pp.ZeroEnd from pp.SensorCal.ZeroMethods.Methods[0]. If ZeroMethods is uninitialized or empty, this may cause IndexOutOfRangeException.

• No Validation of Sensor Serial Number Format:
  Sensors with serial numbers starting with Constants.ISO_CH_ONLY_PREFIX bypass some validation (e.g., sensitivity/capacity checks), but the prefix is not defined in the provided source.

• Error Reporting Inconsistency:
  DTSCSVSensorsParser aggregates errors per line and reports them as a single error string, while DTSCSVTestSetupParser reports errors via IImportNotification.ReportErrors. Error severity and continuation behavior differ between parsers.

• File Lock Detection Limitation:
  CSVFile.IsInUse uses File.ReadAllLines, which may not reliably detect all lock states (e.g., file opened for append-only).