<?xml version="1.0"?>
<ArrayOfFBEvent xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <FBEvent>
    <User>Jaime Lopez</User>
    <Comment>Verified with DataPRO v4.2.324, 4.0.1119
 
Tested:
 
 
With a test setup with the following configuration: 
 
recording mode 
 
circular buffer
 
 
 
Samples per second 
 
400k
 
 
 
Post-trigger seconds 
 
900 (15*60)
 
 
 
 
 
Ran test and recorded 15 min event @ 400ksps
 
Downloaded without failures ✔
 
 
Exported via Ch.10 
 
No error message popup
 
 
logs: 
 
2024-01-15 14:16:27.603 -&gt;  Attribute - SetEventAttribute [00017801], S6A0079, local: 192.168.1.16:57907, Remote: 192.168.4.79:8301 Key: EventHasBeenDownloaded, DataType: UInt32, Length: 4, Overwrite: True Event number: 0, Key: EventHasBeenDownloaded, DataType: UInt32, Length: 4, Value: 1
  
 2024-01-15 14:16:27.606 &lt;-  Attribute - SetEventAttribute [00017801]\[00017801] - (2.9402ms), S6A0079
  
 2024-01-15 14:16:27.606 S6A0079 has completed service DownloadService
  
 2024-01-15 14:16:27.606 DownloadService finished - total time (41m 26s 990ms)
  
 2024-01-15 14:16:27.606 DownloadService.Download: All units are now finished, calling ServiceAvailable
  
 2024-01-15 14:16:27.608 Setting OverallStatus to Cleaning up
  
 2024-01-15 14:16:27.713 DownloadService.Download: All units are now finished, calling ServiceAvailable
  
 2024-01-15 14:16:27.713 DownloadService.Download : calling service available
  
 2024-01-15 14:17:00.783 User Admin navigated to: Export all
  
 2024-01-15 14:17:01.140 DTS File Name: 2024_01_15 13_16 does not match Test Id: S6A .  Using file name as test Id
  
 2024-01-15 14:17:02.620 DTS File Name: 2024_01_15 13_16 does not match Test Id: S6A .  Using file name as test Id
  
 2024-01-15 14:17:02.621 Building test plan
  
 2024-01-15 14:17:02.621 Processing channel: '[S6A0079] CH-01'
  
 2024-01-15 14:17:02.621 Processing test object channel: '?'
  
 2024-01-15 14:17:07.721 User Admin pressed Record_RunButton
  
 2024-01-15 14:17:07.736 DTS File Name: 2024_01_15 13_16 does not match Test Id: S6A .  Using file name as test Id</Comment>
    <StatusString>Closed</StatusString>
    <TimeStamp>2024-01-15T22:22:40Z</TimeStamp>
    <ZendeskNumber>0</ZendeskNumber>
    <Status>Closed</Status>
  </FBEvent>
  <FBEvent>
    <User>Jaime Lopez</User>
    <Comment />
    <StatusString>Assigned</StatusString>
    <TimeStamp>2024-01-15T21:05:49Z</TimeStamp>
    <ZendeskNumber>0</ZendeskNumber>
    <Status>UNKNOWN</Status>
  </FBEvent>
  <FBEvent>
    <User>Dan McFadden</User>
    <Comment />
    <StatusString>Edited</StatusString>
    <TimeStamp>2024-01-02T16:22:41Z</TimeStamp>
    <ZendeskNumber>0</ZendeskNumber>
    <Status>UNKNOWN</Status>
  </FBEvent>
  <FBEvent>
    <User>Dan McFadden</User>
    <Comment />
    <StatusString>ReleaseNoted</StatusString>
    <TimeStamp>2023-12-19T21:22:01Z</TimeStamp>
    <ReleaseNote>Arbitrary length CH10 files can now be exported and much faster.</ReleaseNote>
    <ZendeskNumber>0</ZendeskNumber>
    <Status>UNKNOWN</Status>
  </FBEvent>
  <FBEvent>
    <User>Dan McFadden</User>
    <Comment>4.0/4.2@49983</Comment>
    <StatusString>Resolved</StatusString>
    <TimeStamp>2023-12-19T17:09:48Z</TimeStamp>
    <ZendeskNumber>0</ZendeskNumber>
    <Status>Resolved</Status>
  </FBEvent>
  <FBEvent>
    <User>Dan McFadden</User>
    <Comment />
    <StatusString>Edited</StatusString>
    <TimeStamp>2023-12-18T22:19:16Z</TimeStamp>
    <ZendeskNumber>0</ZendeskNumber>
    <Status>UNKNOWN</Status>
  </FBEvent>
  <FBEvent>
    <User>Dan McFadden</User>
    <Comment>Jerry, John,
 
I've submitted a patch into 4.2 DEVEL for:
 
35627
 CH10 export faults exporting 15m@400ksps
 
This comes from a request to export 400k sps for 15 minutes with 6 different sine waves.  When nate originally tried this DataPRO would throw an exception and fail.
 
 
This was reproducible and there were several issues to be addressed ... The first was that the export would pass the data as arrays of shorts, and there's simply too  much data to be held in an array, nor would you want to pass that much data between functions.
 
There was a function Chapter10File.GetBytes() that would build the entire export into an array of bytes as well then write that all at once to a file.  This also would need to change.
 
The export now writes to the file as a binary stream and no longer keeps huge arrays around.  To get data I added some delegates for getting the next sample rather than passing arrays of data.  This worked and 30+ hours later an export file was produced.  This amount of time was not satisfactory for users, so additional changes had to be made.
 
I did some additional testing, which is documented in the link above a little bit and in chat windows a little bit, but by avoiding getting samples from PersistentChannelInfo and the FileMapViewArray that DataPRO and SLICEWare use, an export that would take 2m 10s went down to 4s. 
 
I changed the export to avoid using PersistentChannelInfo, instead it uses the binary files directly.  The method GetBinaryReaders opens up the files for reading and advances them to the first sample of data in the file. I also wrote a function CloseAndDisposeReaders to close and dispose of all the binary readers when they are finished.  I added the function GetChannelSummaries(Test test) to return all the information I would need from the PersistChannelInfo structure to avoid needing to use it again in the export, so that after that point the persistantchannelinfo structure can be unloaded and thrown away and not needed for the export.
 
This significantly reduces the amount of time needed for the export, however the binary files need to be traversed twice.  The first time it needs to be traversed to determine the min and max ADC for each channel. There's a competing function ComputeMinMaxADC in PersistentChannelInfo, but it's very slow to run in comparison.  The min and max are needed for the TMATS file which is at the very front of the export so this information has to be obtained early on and can't be computed easily while going through the file to produce the output AnalogDataPackets that go into the export.
 
Finally i noticed the progress bar wasn't really hooked up at all to the export, which also made things un-nice.  I added some simple progress by splitting the Compute ADC to the first 50% of export time and the output to the second 50%.
 
I've uploaded the original dataset to
 
\\dts\files\Software\Private\Useful_Test_Files\DataPRO\Data
 
to the files SLICE6Air Siggen Collect_noExports.7z and SLICE6Air Siggen Collect.7z.
 
The export files are huge (3-4GB, so the no exports file is significantly smaller).
 
here are the tests that should be run
 
1) An export of the original All file should complete.  Pre patch it would not
 
2) The output of a smaller file should be essentially identical prepatch to postpatch.  the TMATS file which is at the start of the export won't be identical as it contains some time information from when the export was run, however all the data portions should match, and the TMATS channel information like channel name, min/max values should match.  I would not run this test on the large export dataset, a much smaller dataset should be used, there is an ROI dataset above that can be used
 
3) The export should handle bad data in the dataset without crashing.  If you use a file editor to truncate one of the .chn files to be shorter than it should be the export should fail and there should be logs indicating the issue, but it shouldn't crash datapro.
 
4) Exports should be significantly faster post patch versus pre patch.
 
I think the inefficiency of the FileMapViewArray may cause unnecessary time in other exports and the viewer as well ...
 
If the time exporting is still considered too large, skipping the compute min/max adc or finding a way to perform it inline with the other file traversal would cut time.  It might not cut out 50% of the time, but it would cut out at least 25% of time.</Comment>
    <StatusString>Assigned</StatusString>
    <TimeStamp>2023-12-18T21:45:44Z</TimeStamp>
    <ZendeskNumber>0</ZendeskNumber>
    <Status>UNKNOWN</Status>
  </FBEvent>
  <FBEvent>
    <User>Dan McFadden</User>
    <Comment>the 30H version has been cut down to 20 minutes now.</Comment>
    <StatusString>Edited</StatusString>
    <TimeStamp>2023-12-15T23:48:28Z</TimeStamp>
    <ZendeskNumber>0</ZendeskNumber>
    <Status>UNKNOWN</Status>
  </FBEvent>
  <FBEvent>
    <User>Dan McFadden</User>
    <Comment>the same export, not reading from a binary file, but including byte swap and converting to unsigned takes less than 5s, suggesting most of the time was in the FileMapViewArray ...</Comment>
    <StatusString>Edited</StatusString>
    <TimeStamp>2023-12-15T21:17:09Z</TimeStamp>
    <ZendeskNumber>0</ZendeskNumber>
    <Status>UNKNOWN</Status>
  </FBEvent>
  <FBEvent>
    <User>Dan McFadden</User>
    <Comment>overall time is relatively unchanged when you bypass the byte ordering and signed unsigned, this may not get us much performance increase, will go looking at eliminating FileViewMapArray instead.</Comment>
    <StatusString>Edited</StatusString>
    <TimeStamp>2023-12-15T21:12:47Z</TimeStamp>
    <ZendeskNumber>0</ZendeskNumber>
    <Status>UNKNOWN</Status>
  </FBEvent>
  <FBEvent>
    <User>Dan McFadden</User>
    <Comment>export now completes with new code, however takes 30+ hours.
 
there's a smaller dataset:
 
 
which currently takes
 
 
I'll use this for the performance dataset to test with as it should be quicker to go through iterations rather than the the original dataset.</Comment>
    <StatusString>Edited</StatusString>
    <TimeStamp>2023-12-15T19:30:29Z</TimeStamp>
    <ZendeskNumber>0</ZendeskNumber>
    <Status>UNKNOWN</Status>
  </FBEvent>
  <FBEvent>
    <User>Dan McFadden</User>
    <Comment />
    <StatusString>Edited</StatusString>
    <TimeStamp>2023-12-14T18:05:18Z</TimeStamp>
    <ZendeskNumber>0</ZendeskNumber>
    <Status>UNKNOWN</Status>
  </FBEvent>
  <FBEvent>
    <User>Dan McFadden</User>
    <Comment>will use this dataset to check out performance while exporting the 400ksps data set;
 
 
here's the dataset exported prior to addressing performance.</Comment>
    <StatusString>Edited</StatusString>
    <TimeStamp>2023-12-14T18:03:24Z</TimeStamp>
    <ZendeskNumber>0</ZendeskNumber>
    <Status>UNKNOWN</Status>
  </FBEvent>
  <FBEvent>
    <User>Dan McFadden</User>
    <Comment />
    <StatusString>Edited</StatusString>
    <TimeStamp>2023-12-14T16:03:05Z</TimeStamp>
    <ZendeskNumber>0</ZendeskNumber>
    <Status>UNKNOWN</Status>
  </FBEvent>
  <FBEvent>
    <User>Dan McFadden</User>
    <Comment />
    <StatusString>Edited</StatusString>
    <TimeStamp>2023-12-14T00:19:09Z</TimeStamp>
    <ZendeskNumber>0</ZendeskNumber>
    <Status>UNKNOWN</Status>
  </FBEvent>
  <FBEvent>
    <User>Dan McFadden</User>
    <Comment />
    <StatusString>Edited</StatusString>
    <TimeStamp>2023-12-13T19:11:21Z</TimeStamp>
    <ZendeskNumber>0</ZendeskNumber>
    <Status>UNKNOWN</Status>
  </FBEvent>
  <FBEvent>
    <User>Dan McFadden</User>
    <Comment />
    <StatusString>Edited</StatusString>
    <TimeStamp>2023-12-13T18:53:34Z</TimeStamp>
    <ZendeskNumber>0</ZendeskNumber>
    <Status>UNKNOWN</Status>
  </FBEvent>
  <FBEvent>
    <User>Dan McFadden</User>
    <Comment>it takes about 220s to compute min/max and there's no progress shown during this progress;
 
 
 
 
trying to optimize it  by doing this;
 
                var numSamples = NumberOfSamples;
                 for (ulong sample = 0; sample &lt; numSamples; sample++)
                 {
                     var a = Data[sample];
                     if ( a &gt; max) { max = a; }
                     else if( a &lt; min) { min = a; }
                 }
 
did not result in any performance increase (according to the elapsed counter in visual studio [although it seemed faster to my eyes])</Comment>
    <StatusString>Edited</StatusString>
    <TimeStamp>2023-12-13T18:37:09Z</TimeStamp>
    <ZendeskNumber>0</ZendeskNumber>
    <Status>UNKNOWN</Status>
  </FBEvent>
  <FBEvent>
    <User>Dan McFadden</User>
    <Comment />
    <StatusString>Assigned</StatusString>
    <TimeStamp>2023-12-13T17:46:42Z</TimeStamp>
    <ZendeskNumber>0</ZendeskNumber>
    <Status>UNKNOWN</Status>
  </FBEvent>
  <FBEvent>
    <User>Dan McFadden</User>
    <Comment>reported by Nate:
 
Nathan Brown9:30 AM
 
 
 
 
 
 
 
 
 
 
 
 
15 minutes @ 400ksps * 6 channels exporting... by Nathan BrownNathan Brown9:31 AM
 
15 minutes @ 400ksps * 6 channels exporting to ch10 throws "Stream was too long." exception
 
can you put it up on \\dts\files and I'll c... by Dan McFaddenDan McFadden9:33 AM
 
can you put it up on \\dts\files and I'll check it out?
 
1 Like reaction.1 I haven't looked at the code yet but it doe... by Nathan BrownNathan Brown9:33 AM
 
I haven't looked at the code yet but it doesn't look like it started writing the ch10 file at all yet. Maybe writing directly to disk rather than in memory first?
 
will do by Nathan BrownNathan Brown9:33 AM
 
will do
 
it does need to open all the files as it wo... by Dan McFaddenDan McFadden9:34 AM
 
it does need to open all the files as it works as it outputs all channels each sample tick, but it doesn't need filtering or anything else, so it should be able to handle it, I might need to change the structure it's writing the file out to though, guess I'll find out
 
1 Like reaction.1 I was watching with resource monitor and di... by Nathan BrownNathan Brown9:39 AM
 
I was watching with resource monitor and did see it loop through each .chn file. When that exception popped, it was probably ~3 minutes after it had released all the chn files
 
  image   It'll be here in about 15 minutes... by Nathan BrownNathan Brown9:40 AM
 
 
 
 
 
 
It'll be here in about 15 minutes:
 
\\dts.local\files\!Temp!\!!--Dan--!!\AVMI ch10 Data\SLICE6Air Siggen Collect.zip
 
1 Like reaction.1 thanks, I'll check it out shortly by Dan McFaddenDan McFadden9:40 AM
 
thanks, I'll check it out shortly
 
has context menuParagraph
 
 
 
has context menu</Comment>
    <StatusString>Opened</StatusString>
    <TimeStamp>2023-12-13T17:46:42Z</TimeStamp>
    <ZendeskNumber>0</ZendeskNumber>
    <Status>Opened</Status>
  </FBEvent>
</ArrayOfFBEvent>