Figure: Neuroarchiver Tool on MacOS.
Note: This manual applies to Neuroarchiver 64 with LWDAQ 7.6+. See Changes for new features.
The Neuroarchiver is a program written in TclTk. It is available in the Tool Menu of the LWDAQ Program. The Neuroarchiver.tcl source code is in the Tools directory of the LWDAQ distribution. The Neuroarchiver works with the Recorder Instrument and LWDAQ data acquisition hardware such as the Data Receiver (A3018). The Data Receiver records signals from one or more Subcutaneous Transmitters (A3013 or A3019). The Neuroarchiver downloads the signals from the Data Receiver over the Internet with a LWDAQ Driver acting as an intermediary. The Neuroarchiver acquires raw data continuously and saves it to disk. At the same time, the Neuroarchiver can read raw data from disk, display the recorded signals on the screen, and execute user-defined processing and classification. The Neuroarchiver's recording and playback are independent. We can play back data as we record it, or we can play back previously-recorded data while at the same time recording new data.
The original motivation behind the design of the Subcutaneous Transmitter was to detect epileptic seizures in rats. The Neuroarchiver attempts to provide automatic event detection in telemetric signals. It's event-detecting process consists of four steps.
If we consider a playback interval of 4 s, we see that recording the data takes 4 s. We find on our laptop that extraction of the voltages and spectrum of 4 s of data with four active channels takes a fraction of a second if we disable the plots, and around 1 s if the plots are enabled. Processing of the playback interval to produce its characteristics takes tens of milliseconds. If we are playing back the data only for the purpose of processing, we usually speed things up by turning off the plotting. When we later apply an event-detection program to the characteristics, each playback interval is represented by ten or twenty numbers, and execution is fractions of a millisecond. Thus the creation of the characteristics file is intensive, because it requires playback of the data. But event detection applied to the characteristics file is fast. By choosing the characteristics well, and keeping the characteristics file on disk, we can hope to avoid repeating the playback, but try out many different types of event detection while perfecting our process.
The Configure button opens an array of configuration parameters. The Help button offers some basic help. The Pick buttons allow us to select files named in the adjacent text label. The Recorder file has two choose buttons: Pick File, which we use to select an existing archive to which we want to add data, and Pick Directory, which we use to select a directory in which to create a new archive.
In the figure above, the Recorder portion of the Neuroarchiver is downloading data in half-second intervals and storing them to an archive whose name appears to the right of the Recording Archive label. The Recorder state is Record. A white background to the recorder state means the Recorder is waiting for the Player to finish its processing. A yellow background means the Recorder is waiting for data from the data receiver. Every 3600 seconds the Recorder will produce a new archive file. The archive files are in the NDF (Neuroscience Data Format).
Meanwhile, the Player is reading from a different archive, which contains data from multiple Subcutaneous Transmitters (A3019D) implanted in live rats. We can see the active transmitters int the Activity label. Each channel number is followed by a colon and the number of messages from this channel in the raw data for the playback interval. The clock channel (channel 0) should always receive 128 MSP (messages per second), or 512 in a 4-s interval. A transmitter running at 512 MSP will provide up to 512 messages in a 1-s interval.
The Processor is enabled, so that the processing script will be applied to each playback interval. The result is the lines we see on the screen. These lines are not currently being saved to disk, however, because the Save button is not checked.
The Value vs. Time plot shows the signal voltages during the playback interval. The Amplitude vs. Frequency plot shows the frequency spectrum of the signals during the playback interval. We choose which channels will be plotted, transformed, and processed with the Select entry box. The string "1 2 3" would select channels 1, 2, and 3 only. An asterisk (*) selects all available channels. We discuss channel selection in more detail below. Even the results of processing, shown in green, are restricted to the selected channels.
The Player state is Play. A white background means the Player is waiting for the Recorder to finish acquiring data. A green background means that the Player is analyzing messages. A yellow background means the Player is waiting from data to be added to its archive. The Player will be waiting regularly if it is playing Recorder file, because it must wait until the required data is added. When the orange background appears behind the Player state, the Player is jumping to a new archive or to a new point within an archive.
Before we describe how to run the Neuroarchiver, make sure that your Subcutaneous Transmitter system is set up for recording. Follow the set-up instructions here. Once the LWDAQ is working, with connection to your LWDAQ Driver, select the Neuroarchiver from the Tool menu.
Before you acquire data with the Neuroarchiver from your own hardware, you must set up the Recorder Instrument to be the source of data. You are welcome to use the Recorder Instrument's default settings, which will acquire raw data from our demonstration hardware. But if you do, please be sure to stop recording from our demonstration stand as soon as you have finished your exercises, so that someone else can use the demonstration stand's hardware for their own tests.
Here is how you start recording and simultaneous playback with the Neuroarchiver. The Neuroarchiver uses the Recorder Instrument to acquire new data, so the first steps are to set up the Recorder Instrument to point to your Data Receiver (such as the A3018).
Look at your data recorder. The EMPTY light should be flashing regularly. If it is not flashing, your Neuroarchiver is not acquiring data as fast as the data is being recorded. This failure to keep up with the pace of recording may be due to your network connection being too slow to transport the data as it is created.
Once you get the recording and playback working, you can try out various values of recording interval and playback interval. For the most stable operation with up-to date signal display, the recording interval should be half the playback interval. In stable operation, the Player is waiting for the Recorder to save data to disk. When the data is available, the Player displays it. While it waits, the Player state indicator is yellow.
You can look through previously-recorded archives even while you are recording a new archive. Stop the simultaneous playback and select a new archive. If you want to see an overview of an entire archive, select it in the Player and press the Overview button.
The two plots are drawn during playback of recorded data. Each plot has its own Enable check-box. Plotting can slow down playback. Disable the plots by un-checking their enable check-boxes. Playback will speed up.
The Value vs. Time graph, which is on the left of the Neuroarchiver panel, shows signal voltage variation during the most recent playback interval. In the example display we see four traces. This plot is identical to the one generated by the Recorder Instrument. The traces are color-coded. For a table of channel numbers and their colors, see the Message Display section of the Recorder Instrument Manual, or this picture. The vertical axis is voltage, measured in ADC counts. Each transmitter converts its analog input into a sixteen-bit value. Sixteen-bit values run from 0 to 65535. In the example, the plot spans the range 0 to 65535. The bottom of the range is zero because the value of v_offset is 0. The top of the range is 65535 counts above the bottom because v_range is 65535. To convert between ADC counts and voltage at the transmitter input, consult the transmitter manual.
Example: The A3013A transmitter manual has a section called Analog Inputs. Here we see that the gain of the transmitter from analog input to the ADC is 300. The voltage range 0 to 65535 corresponds to voltages 0 V to VBAT at the ADC input. For most of a transmitter's operating life, VBAT is 2.7 V, so each ADC count is 140 nV at the X input. The amplifier AC-couples the X input, placing its average value at 1.8 V. The dynamic range for signals at X is −6 mV to +3 mV. We can deduce the battery voltage, VBAT by looking at the average value of the signal in the plot. If A is the average value of X as a fraction of 65535, we have VBAT = 1.8 V / A. From the signal present in archive M1259065886.ndf, we estimate A is 0.64 for channels No1 and No2, 0.70 for channel No14, and 0.8 for channel No6. This implies VBAT of 2.8 V for No1 and No2, 2.6 V for No14, and only 2.2 V for No6. We can compare these voltages to this graph to estimate how much more operating life each transmitter has left.
Example: The A3019A transmitter has a lower gain and therefore higher dynamic range than the A3013A. The gain of the A3019A is 100, so that each ADC count represents 400 mV. The dynamic range is around 20 mV.
The horizontal axis in the Value vs. Time graph is time. The full range from left to right covers the most recent playback interval. This interval is shown in the Interval selector beneath the graphs.
We can obtain a close-up of fluctuations in the analog signal by entering AC for v_coupling and changing v_range to a smaller value. With AC coupling, the Neuroarchiver places the average value of the signal half-way up the display.
The Amplitude vs. Frequency graph is the discrete fourier transform of the reconstructed signal. The transform amplitude range is zero to a_range in ADC counts. The Player takes the discrete fourier transform of the interval data for each channel. It calculates all terms in the transform and plots those between f_min to f_max. The discrete fourier transform dictates frequency step from one discrete component to the next. We have f_step = 1/p, where p is the play interval. The highest frequency component in the transform is at half the transmitter's message frequency. The transform applies to the reconstructed signal. For a 512 MPS transmitter, the highest frequency component in the transform will be 256 Hz. If we set the range of the frequency plot outside the range zero to one half the sampling frequency, the spectrum will be blank.
Aside: As we describe in the Recorder Instrument Manual, the reconstructed signal will always contain messages at exactly the transmitter's nominal frequency, regardless of how many messages we lost in reception. We calculate the transform using an fast fourier transform algorithm. This algorithm requires a perfect power of two number of samples as its input, in order to allow its divide and conquer method to operate with perfect symmetry upon the problem. All our transmitters operate at a frequency that is a perfect power of two, so choosing playback intervals that are a power-of-two fractions of multiples of one second will always give us a number of samples that satisfies our algorithm. It is possible to turn off reconstruction in the Neuroarchiver by setting enable_reconstruct to 0. If we turn off reconstruction, the Neuroarchiver will perform an abbreviated reconstruction for the algorithm by adding dummy messages to or subtracting excess messages from the raw messages sequence.
Example: With amplitude range 1000 counts, each vertical division is 100 counts. Suppose our sample rate is 512 SPS. We set f_min to 0 Hz and f_max to 256 Hz so that we can see the entire discrete fourier transform of the 512 samples taken in the 1-s play interval. The frequency step is 1 s because the play interval is 1 s. If we switch the play interval to 4 s, the Neuroarchiver will set the frequency step to 0.25 s.
If we click the Log checkbox, the frequency axis will become logarithmic, with lines marking the decades in the traditional fashion.
Aside: We do not provide logarithmic display for the amplitude, although we could easily do so if it were to prove useful. Furthermore, we feel that the logarithmic frequency display is not particularly useful because the fourier transform components are distributed in uniform frequency steps instead of logarithmic steps.
We describe the generation and adjustment of the frequency spectrum in more detail below.
The Neuroarchiver window displays the names of four files: the recording archive, the playback archive, the processing script, and the event list. We can select these files from our own file system using the Pick buttons beside each file name. The Recorder has a Pick File and a Pick Directory button. We begin recording by pressing the Reset button, which causes the Recorder to create a new archive for us in the same directory as the previous archive.
The Neuroarchiver stores transmitter messages in NDF (Neuroscience Data Format) files. It performs no processing upon the messages as it stores them to disk. What appears in the NDF file is exactly the same sequence of messages that the Data Receiver stored it its memory. Thus we have the raw data on disk, and no information is lost in the storage process.
An NDF file contains a header, metadata string, and a data block to which we can append data at any time without writing to the header or the data string. We define the NDF format in the Images section of the LWDAQ Manual. The Neuroarchiver manipulates NDF files with a NDF-handling routines provided by LWDAQ. These routines are declared in LWDAQ's Utils.tcl script. You will find them described in the LWDAQ Command Reference. Their names begin with LWDAQ_ndf_.
All archives created by the Neuroarchiver receive a name of the form "Mxxxxxxxxxx.ndf", where the ten x's give the number of seconds since 1900 hours on 31st December, 1969. This number is the standard UNIX timestamp. We can obtain it at any time in a Tcl script with the command clock seconds. From the name of each file, we can determine the time, to within a second, at which its first clock message occurred. From there we can count clock messages and determine the time at which any other part of the data occurred.
The Recorder stores data in NDF archives and the Player reads the NDF archives to extract voltages and calculate spectra. When the Player reaches the end of an archive, it looks for a newer archive in the same directory and starts playing that one immediately afterwards. Thus if we are playing the archive that is being recorded, the Player will play the fresh data from the expanding archive until the Recorder starts a new archive, at which time the Player will switch to the new archive automatically. If we are playing old archives, the Player will still move from the end of one to the start of the next, even if the next is unrelated to the first. Thus we can go through a collection of archives that are from different experiments and different times, and apply processing to extract characteristics from all the archives.
The Recorder and Player both provide a Metadata button. This button opens up a text window that displays the metadata and allows us to add comments and save metadata to disk. The comments in an archive's metadata can remind us of what the file contains. The generic names of our archives don't help much when it comes to identifying particular experiments. So the Player provides a List button that brings up a new window with a list of all the NDF files in the same directory as the player archive, and displays all their metadata comments.
In addition to NDF files, the Neuroarchiver works with two classes of text files. The first are Processing Scripts. These are TclTk programs that the Player will apply to the signals in each playback interval. The second are Event Lists. These are lists of events detected in recorded signals that the Player uses to navigate between events. We select these files with Pick buttons, but the Neuroarchiver provides no way to edit such files. We assume you have a text editor on your computer.
The data acquired by the Neuroarchiver takes the form of a list of data recorder messages, as we describe elsewhere. In general, the data will contain values from one or more channels. The Neuroarchiver selects which channels to display, transform, and store to disk using its channel_select parameter.
In its simplest form, channel_select is a single "*" character. With channel_select set to "*", the Neuroarchiver looks through the playback interval data and counts how many messages it contains from each of the 16 possible subcutaneous transmitter channels: No0 through No15. Of these, No0 is the clock message channel and No15 is the slow data channel. Transmitters like the A3013A use the fast data channels, No1 through No14. If a channel contains more than activity_threshold messages, the Neuroarchiver considers it active. It plots all active channels and lists them in the Channel Activity string in the format id:qty, where id is the channel number and qty is the number of messages. You will find the activity_threshold parameter in the configuration array (press Configure).
We can select particular channels with a specific channel_select string. We can enter "1 2 6 14" and the Neuroarchiver will attempt to display these channels, even if they have very few messages. We can specify the nominal sampling frequency and scatter extent for each channel. For a description of sampling frequency and scatter extent, see Analysis section of the Recorder Instrument Manual. If we list the channel numbers on their own, the Neuroarchiver assumes a the frequency and scatter given by default_frequency and default_scatter. We specify channel number, frequency, and scatter with three numbers in the form c:f:s. Thus "5:512:8" means channel 5 with sampling frequency 512 SPS and scatter extent 8.
The sampling frequency and scatter extent are used by the Neuroarchiver when it reconstructs an incoming message stream. The Neuroarchiver uses its clocks_per_second and ticks_per_clock parameter to convert samples per second into a sample period in units of data recorder clock ticks. The Neuroarchiver can then go through a channel's messages and identify places where messages are missing, and eliminate bad messages that occur in the message stream at random times.
By default, the Neuroarchiver applies reconstruction to all data during playback. But we can disable the reconstruction by setting enable_reconstruct to zero in the configuration array. We sometimes disable reconstruction so we can get a better look at bad messages and other reception problems.
The Neuroarchiver uses the Recorder Instrument to obtain live data. First we set up the Recorder Instrument to read out streams of messages from a data-recording device, then we open the Neuroarchiver to store the live data. The Neuroarchiver does not display new data until after it has been stored to disk, so display and processing are not part of data acquisition.
To capture live data, open the Recorder Instrument and configure it to read our data out of our data recorder. The Neuroarchiver will simply call the Recorder Instrument's data acquisition procedure when it captures new live data. We don't have to leave the Recorder Panel open after we set it up for data Recording, but we can leave it open if we like. By default, the Neuroarchiver turns off the Recorder Instrument's plotting and analysis of the incoming data. But we can turn on the plotting and analysis by setting enable_recorder_analysis to 1 in the Neuroarchiver's configuration panel, which we can open with the Configure button. With the plots turned off, we will see the raw data displayed as gray-scale pixels in the Recorder Instrument.
The only thing that passes from the Recorder Instrument to the Neuroarchiver is the raw data acquired from the data acquisition hardware. The Recorder Instrument has a data acquisition parameter called daq_num_clocks. When we instruct the Recorder Instrument to acquire new data, it acquires a block of messages with exactly this number of clocks. The Recorder Instrument makes sure that first message in the block is always a clock message. The Neuroarchiver calculates daq_num_clocks from record_interval, which has units of seconds. In the example shown above, the recording time interval is 1.0 s, and is shown in the menu-button to the right of the Recorder controls.
We can view the recording file's metadata with the Recorder's Metadata button.
The NDF format contains a header, a metadata string, and a data block. Transmitter messages and clock messages are stored by the Recorder in the data block. New data is appended to the data block without any alteration of existing data. The metadata string has a fixed space allocated to it in the file, but is itself of variable length, being null-terminated. We can view, edit, and save metadata strings with the Metadata buttons. When the Recorder creates a new file, the metadata contains a single multi-line comments. The comment is delimited by an xml "c" tags.
<c> Date Created: Thu Sep 2 15:45:59 2010. Creator: Neuroarchiver 45, LWDAQ_7.4.4. Host: dyn-129-64-201-94.wireless.brandeis.edu </c>
We can enter our own comments in the metadata of any archive using the Metadata button. We can add our own comments to the existing comment field written by the Recorder at the time of creation, or we can add our own comment fields delimited by "c" tags. The List button opoens a list window that provides us with a list of all the archives in the directory of the playback archive, along with all the comments from their metadata strings. Thus we can use metadata comments to describe the contents and origin of our archives, and then view these comments later.
The Neuroarchiver performs all its signal reconstruction and plotting when its Player reads data from disk. Each plot has its own enable checkbox. If we want the Player to calculate interval characteristics with a processing script, we can accelerate the processing by turning off the plots.
If we check the Verbose box on the Neuroarchiver, the Player will report on its reading and processing of data. We will see the loss in each channel, and the results of reconstruction and extraction of message from the playback interval's data. In the example display, we can see the Player telling us that it extracted two channels and reconstructed the other two.
We can navigate through the playback archive by entering a new Time value and pressing Step. We will see the end time of the archive, which is the archive length in terms of the number of clock messages it contains, on the far right of the Player. The Back button steps back one playback interval.
The Play button starts repeated execution of the Step command, so that the Player proceeds through the archive while we watch or wait. The Stop button stops Play. The Repeat button causes the Player repeat the processing and display of the current playback interval. We use the Repeat button when we change the plot ranges or processing script so as to re-display and re-calculate characteristics of the same interval.
The five player buttons do not allow us to navigate between archives, other than that the Player will automatically move from the end of one archive to the beginning of the next when it is playing. If we want to move from one event to another within and between archives, we can use an event list. The Player provides Next, Go, and Previous buttons, as well as an event index, to allow us to navigate through an event list.
We select an archive for playback with the Playback Archive Pick button. If we want to get a list of available archives, we press the List button. The Neuroarchiver will open a new window and display a list of the archives available in the same directory as the current playback archive. Along with the list of names will be the comment fields from the archive metadata strings. the list window provides three buttons for each archive: Step, Metadata, and Overview. These allow us to step directly into the start on an archive, edit the metadat, or jump to an overview of its contents.
The Overview button opens the Player archive and creates a condensed view of its entire contents. The overview opens in a separate window and takes a few seconds to appear. The Neuroarchiver selects num_samples random points from the archive and divides them up into separate channels. It uses those in channel No0 to calculate the approximate time of the transmitter channels. The result is a plot that gives a good representation of the archive contents, but not an exact representation.
The Plot button allows you to re-plot the overview with new time and value ranges. You can select channels with the Select string. The Export button saves the displayed data to disk. Each selected channel will receive its own file named En.txt, where n is the channel number. The export file will appear in the same directory as the archive. If you want more resolution in your overview, increase the number of samples the overview makes of the archive for its display. Note that the number of samples is not the same as the number of points in each signal plot. The activity label tells us the distribution of the samples among the channels.
We calculate the spectrum of each channel signal using the lwdaq_fft routine, which is available to us at the LWDAQ command line. The lwdaq_fft routine takes the sequence of sample values produced by reconstruction and returns the complete discrete fourier transform. If we pass N terms to the transform, we get N/2 terms back.
The routine insists upon a number of samples that is an exact power of two. So we can pass it 16, 32, 256, 512, or 1024 samples. Otherwise the routine gives us an error. Signal reconstruction ensures that we have a suitable number of samples. If we turn off reconstruction by setting enable_reconstruct to zero, the Neuroarchiver adds or subtracts samples to or from the signal so as to satisfy the fourier transform's requirements.
When a signal's end value differs greatly from its start value, the fourier transform sees a sharp step at the end of what it assumes is a periodic function represented by the signal interval. Such a step generates power at all frequencies of the spectrum, rendering the spectrum less useful for detecting events such as epileptic seizures. The Neuroarchiver applies a window function to the signal before it applies the fourier transform. The window_fraction element in the Neuroarchiver's configuration array gives the fraction of the signal that should be subject to the window-function at the start and at the end of the sequence of available samples. We like to use window_fraction 0.1 for EEG signals. The window function is provided as an option in the lwdaq_fft routine.
Data from wireless transmitters often contains bad messages arising from interference and noise. Signal reconstruction attempts to eliminate these messages, and almost always succeeds in doing so, but we find that we still get several bad messages per hour on each signal channel. These appear as spikes, or glitches, in the data. The Neuroarchiver can eliminate almost all of these bad messages with a glitch filter, which is provided as an option by lwdaq_fft. The glitch_threshold element in the configuration array specifies a threshold for glitch recognition. Any sample that differs by more than glitch_threshold from the previous sample will be over-written by the previous sample. This algorithm works well unless the first sample in the interval is itself a bad message, in which case the entire interval will most likely be over-written by the bad message value.
With glitch_threshold = 10000, we observed several glitches per hour from implanted A3013A transmitters, each lasting one or two samples. With 4-s intervals each containing 2048 samples, we expect to lose one 4-s interval because of a poorly-placed glitch once every two thousand hours, which is less than 4 s of lost data during the operating life of the transmitter.
In each playback step, the Neuroarchiver goes through each channel selected by channel_select and performs reconstruction, spectrum-calculation, plotting, and processing. The plotting and processing are optional. We enable processing with the enable processing checkbox. The processing reads the processor from disk. The processor must be a proper TclTk script. The Neuroarchiver executes the script once for each selected channel.
If you want to learn how to program in TclTk, so that you can write your own processors, we recommend Practical Programming in TclTk. Otherwise, you can consult About TclTk and the TclTk Manual. The language is interpreted rather than compiled, and the interpreter is available on all operating systems. Thus our LWDAQ software, and any scripts you write in TclTk, will work in MacOS, Windows, and Linux equally well.
The processing script has access to the Neuroarchiver's configuration and information arrays with the config(element_name) and info(element_name) respectively. The configuration parameters are ones the user is free to modify. The info parameters are a mixture of parameters that are too numerous to list in the configuration array and others that the user should not change. The processing script also has access to several temporary variables. We list some of the most useful variables in the following table.
| Variable | Contents |
|---|---|
| num_clocks | The number of clock messages in the current playback interval |
| result | The processing results string, integers are always channel numbers |
| config(play_file) | The NDF archive being played back |
| config(play_time) | Seconds from archive start to interval start |
| config(enable_vt) | Voltage-time display is enabled |
| config(processor_file) | The processing script file |
| config(channel_select) | The channel-selection string, if * then all channels chosen |
| config(play_interval) | The playback interval in seconds |
| info(channel_num) | The number of the channel just reconstructed and transformed |
| info(num_received) | The number of messages received in this channel during this interval |
| info(num_messages) | The number of messages in the reconstructed signal |
| info(loss) | The signal loss as a percentage. Subtract from 100% to obtain reception efficiency. |
| info(signal) | The reconstructed signal as sequence of timestamps and values |
| info(spectrum) | The transform as sequence of amplitudes and phases |
| info(f_step) | The separation in Hertz of the transform components, equal 1/play_interval |
| info(bp_n) | The baseline power of channel n |
The characteristics are stored in the result string. Any word or number can be added to the characteristics of each channel, except that only channel numbers may be written to the string as integers. Subsequent analysis is able to separate the characteristics of the various channels by looking for the integers that delimit the channel data. If we want to store a value 4 as a characteristic, we can write it as 4.0.
Other elements of the configuration array we can find by pressing the Configure button in the Neuroarchiver window. Each is available with config(element_name) in the processing script. The information array elements we will have to seek out at the top of the Neuroarchiver script itself, where each is described in the comments.
If we select four channels for playback, the processing script will be called four times. Each time the Neuroarchiver calls the script, all variables that are specific to individual channels, such as loss, num_received, signal and spectrum, will be set for the current channel. We obtain the current channel number through the channel_num parameter.
The first time the processing is called, the result string is empty. Each call to the processing should append some more values to the result string. After the final call to the processing script, if the Neuroarchiver sees that result is not an empty string, it prints it to the text window. If the Save box is checked, it appends the string to the a characteristics file. The Processor constructs the name of this file from the name of the archive and the processing script. Script P.tcl, when applied to archive M1234567890.ndf produces a characteristics file called M1234567890_P.txt.
The following script records the reception efficiency for each active channel. This allows us to plot message reception versus time by importing the characteristics file into a spreadsheet.
append result "$info(channel_num) [format %.2f [expr 100.0 - $info(loss)]] "
Once the anlaysis has been applied to all active channels, the Player checks the result string. If the string is not empty, the Player adds the name of the play file and the play time to the beginning of the string. These two pieces of information apply equally to all channels, and are essential characteristics for event detection.
Because the script is TclTk, it can do just about anything that TclTk can do. In theory, it can e-mail the finished result string to us, or upload it over the network to a server. Most processor scripts produce characteristics files through use of the result string. But we can also use processing to export signals or spectra to disk.
The reconstructed signal is available in info(signal). The signal takes the form of a sequence of numbers separated by spaces. Each pair of numbers is the time and value of the signal. The time is in clock ticks from the start of the playback interval. The value is in sixteen-bit ADC counts. The timestamps are twenty-four bit numbers that give the number of data receiver ticks since the start of the playback interval. A twenty-four bit number is up to 16.8 million, and the tick frequency in the A3018 data receiver is 32.768 kHz. The maximum interval we can cover with these timestamps is 512 seconds. We usually specify intervals between 0.1 and 10 s. The sample values are sixteen-bit un-signed numbers.
The discrete fourier transform of the signal is available in info(spectrum). The spectrum is a sequence of numbers separated by spaces. Each pair of numbers is an amplitude and a phase. The pairs are numbered 0 to n−1, where n is the number of samples in the signal, available in num_messages. The k'th pair of numbers describes the frequency component with frequency k×f_step. The amplitude is in sixteen-bit ADC counts and the phase is in radians.
The following processing script illustrates how to manipulate the individual components of the signal spectrum. We can manipulate individual sample values in the same way. The script calculates the total signal power in the 2−40 Hz frequency range and appends the channel number and the power measurement to the result string. If the voltage-time plot is enabled, the processing continues by plotting the 2−40 Hz band-pass filtered signal on the screen. It obtains the band-pass filtered signal by copying the original fourier transform, setting the components outside 2−40 Hz to zero, and obtaining the inverse transform.
set band_lo 2
set band_hi 40
set band_power 0.0
set f 0
foreach {a p} $info(spectrum) {
if {($f >= $band_lo) && ($f <= $band_hi)} {
set band_power [expr $band_power + ($a * $a)]
}
set f [expr $f + $info(f_step)]
}
append result "$info(channel_num) [format %.1f $band_power] "
if {$config(enable_vt)} {
set new_spectrum ""
set f 0
foreach {a p} $info(spectrum) {
if {($f >= $band_lo) && ($f <= $band_hi)} {
append new_spectrum "$a $p "
} {
append new_spectrum "0 0 "
}
set f [expr $f + $info(f_step)]
}
set new_values [lwdaq_fft $new_spectrum -inverse 1]
set new_signal ""
set timestamp 0
foreach {v} $new_values {
append new_signal "$timestamp $v "
incr timestamp
}
Neuroarchiver_plot_signal [expr $id + 32] $new_signal
}
The mean square value of the filtered signal is equal to the total power we obtain by adding together the squared fourier components. If the frequency band does not include zero, the filtered signal will have average value zero, because its zero-frequency, or DC, component has been removed. In order to see the filtered signal overlaid upon the original signal, we must use AC coupling in the display.
We provide the Neuroarchiver_band_power command to do most of the work in the above code for us, and a bit more. The routine makes sure that the DC component of the filtered signal is included before plotting, so the filtered signal is always overlaid upon the original signal in the display. You will find Neuroarchiver_band_power defined in the Neuroarchiver.tcl program. The procedure takes four parameters: the low frequency, the high frequency, a boolean flag to plot the filtered signal on the screen, and a boolean flag to put the filtered signal in info(values). When the routine plots the filtered signal, it picks a color automatically. The result looks like this (4-s transients filtered to 2-160 Hz) and this (1-s seizure filtered to 2-160 Hz).
append result "$info(channel_num) [format %.2f [expr 100.0 - $info(loss)]] " set tp [expr 0.001 * [Neuroarchiver_band_power 0.1 1]] set sp [expr 0.001 * [Neuroarchiver_band_power 2 20 $config(enable_vt)]] set bp [expr 0.001 * [Neuroarchiver_band_power 40 160 0 1]] append result "[format %.2f $tp] [format %.2f $sp] [format %.2f $bp] "
This script calculates power in three bands: transient, seizure, and burst. The result string shows the power in all bands. Power in the first band is a feature of step-like changes from problematic electrodes. Power in the second band is high during epileptic seizures. Power in the third band occurs during wave bursts. The script plots the second band when the value plot is enabled and leaves the third band values in info(values) for the processor script to manipulate.
Here are some further examples.
# Export signal values to text file. Each active channel receives a file
# En.txt, where n is the channel number. All values from the reconstructed
# signal are appended as sixteen-bit integers to separate lines in the file.
# Because this script does not use the processing result string, the Player
# will not create or append to a characteristics file.
set fn [file join [file dirname $config(processor_file)] "E$info(channel_num)\.txt"]
set export_string ""
foreach {timestamp value} $info(signal) {
append export_string "$value\n"
}
set f [open $fn a]
puts -nonewline $f $export_string
close $f
# Export signal spectrum, otherwise similar to above value-exporter. The
# script does not use the result string, and so produces no
# characteristics file. Instead of appending the spectrum to its output
# file, each run through this script re-writes the spectrum file.
set fn [file join [file dirname $config(processor_file)] "S$info(channel_num)\.txt"]
set export_string ""
set frequency 0
foreach {amplitude phase} $info(spectrum) {
append export_string "$frequency $amplitude\n"
set frequency [expr $frequency + $info(f_step)]
}
set f [open $fn w]
puts -nonewline $f $export_string
close $f
# Calculate and record the power in each of a sequence of contiguous
# bands, with the first band beginning just above 0 Hz. We specify the
# remaining bands with the frequency of the boundaries between the
# bands. The final frequency is the top end of the final band.
append result "$info(channel_num) "
set f_lo 0
foreach f_hi {1 20 40 160} {
set power [Neuroarchiver_band_power [expr $f_lo + 0.01] $f_hi 0]
append result "[format %.2f [expr 0.001 * $power]] "
set f_lo $f_hi
}
# Here's another way to obtain power in various bands. We specify the
# lower and upper frequency of each band.
append result "$info(channel_num) [format %.2f [expr 100.0 - $info(loss)]] "
foreach {lo hi} {1 3.99 4 7.99 8 11.99 12 29.99 30 49.99 50 69.99 70 119.99 120 160} {
set bp [expr 0.001 * [Neuroarchiver_band_power $lo $hi 0]]
append result "[format %.2f $bp] "
}
Processors that assist with event detection, such as classification processors, are far longer than our examples. The ECP1 processor, for example, is 170 lines long.
Suppose we want to process thousands of hours of data from a dozen transmitters stored on disk. We can open the Neuroarchiver and start processing, but we will have to wait hundreds of hours, and our computer screen will be occupied by the Neuroarchiver display. On Linux, Unix and MacOS, however, we can run the Neuroarchiver without graphics as a console application or as a background process with no console at all. We can take our list of archives and divide their processing among a cluster of computers, with a separate instance of the Neuroarchiver running on each computer. With no graphics, the processing is over twice as fast, and with ten computers, another ten times as fast, so batch processing can be one or two orders of magnitude faster than standard processing on your laptop.
To set up batch processing, start by consulting the Run In Terminal section of the LWDAQ Manual. The idea is to invoke LWDAQ from the command line using the lwdaq shell script that comes with every LWDAQ distribution. You will need LWDAQ 7.5+ and Neuroarchiver 50+. The following command invokes LWDAQ as a background process, executes a configuration script, and passes the name of an archive and a processor into LWDAQ.
lwdaq --no-console config.tcl M1288538199.ndf processor.tcl
The archive is the file ending in NDF. It contains binary data recorded from the subcutaneous transmitters. The processor.tcl file is a text file containing a processor script to create the lines of a characteristics file. The config.tcl file is a configuration script. Here is an example configuration script.
# The following LWDAQ configuration script opens the Neuroarchiver, # selects an archive and applies a processor to 4-s playback intervals # while saving the characteristics to disk. At the end of the archive, # the Neuroarchvier stops and LWDAQ quits. LWDAQ_run_tool Neuroarchiver.tcl set Neuroarchiver_config(play_file) [lindex $LWDAQ_Info(argv) 0] set Neuroarchiver_config(processor_file) [lindex $LWDAQ_Info(argv) 1] set Neuroarchiver_info(play_control) Play set Neuroarchiver_config(play_interval) 4 set Neuroarchiver_config(enable_processing) 1 set Neuroarchiver_config(save_processing) 1 set Neuroarchiver_config(play_stop_at_end) 1 LWDAQ_watch Neuroarchiver_info(play_control) Idle exit Neuroarchiver_play
The configuration script sets up the Neuroarchiver to read through the archive in 4-s intervals, creating a characteristics file in the manner described above. When it's done with the archive, it stops and terminates. (The LWDAQ_watch command does the termination.) We assume that that batch job manager will keep track of which analysis processes are still running, and add new ones as the previous ones terminate.
When a processor relies upon baseline power calibration, such as an event classification processor, we must establish baseline calibration before we start recording characteristics to disk. The configuration script can set up the Neuroarchiver to apply the processor to a calibration period, and then go back to the start of the archive to process the archive and record characteristics to disk. An example script to establish basline calibration is Restart_Config.tcl.
Once we have applied processing to our data archives to produce characteristics files, we can look for events, calculate average characteristics, or determine summary information by analyzing the characteristics files. We call this step analysis. When the analysis detects events, we call the analysis program an event-detector.
The Seizure-Detector, Mark I (SD1) script is an example of an event-detector written in TclTk that we can run in the LWDAQ Toolmaker. The script looks through the characteristics produced by the TPSPBP processor and detects epileptic seizures by examining the development of seizure-band power in the absence of transient-band power.
The Power Band Average (PBA) script calculates the average power in a sequence of frequency bands during consecutive intervals in time. You specify the length of these intervals in the script, in units of seconds. So they could be minutes to hours or days. You run the script in the Toolmaker and specify any number of characteristics files with the file browser. Cut and paste from the window to plot in Excel.
The Average Reception (RA) script calculates average reception during consecutive intervals of time. It is similar to the Power Band Average script in the way it reads in characteristics files one after another and prints its results to the screen.
The Reception Failure (RF) script looks for periods of reception failure and writes an event list to the Toolmaker execution window. Cut and past the list into a file to make an event list the Neuroarchiver can step through.
The Bad Characteristics Sifter (BCS) script goes through characteristics files and extracts those corresponding to one particular channel, provided that the characteristics meet certain user-defined criteria, such as minimum or maximum power in various frequency bands.
We present the development of seizure detection using interval analysis in Seizure Detection. The Neuroarchiver's built-in Event Classifier provides analysis that compares intervals with reference cases to detect and identify events in recorded signals.
The power of a signal is almost always essential for reliable event detection. Only when the power exceeds a certain threshold do we declare an unusual event to have taken place. The sensitivity of our transmitters, however, varies from one sensor to the next. In once case, two implanted A3019D transmitters differed in their sensitivity to EEG by a factor of ten (No11 and No4 from M1300920651 at ION). For each sensor we need a baseline power by which we can divide its absolute signal power to obtain a normalized power measurement. The Neuroarchiver provides the info(bp_n) parameters to store the baseline power for each channel. We can view and edit the current baseline powers with the Baselines button.
Processor scripts like ECP1 look for a minimum in signal power, and use this as the baseline. They increase the baseline power by a tiny amount every interval, so that the processing can accommodate a decrease in sensitivity. Before we begin an experiment with such a processor, we must be sure to run it for a while to determine the baseline powers, and then start our processing in earnest.
The Event Navigator, which is the section of the Neuroarchiver that allows us to navigate through event lists, will try to deduce the baseline power at the time of an event, and set the baseline power for the event channel before it jumps. The Event Classifier provides a list of events, each of which contains the baseline power at the time of the event, and the Event Classifier will always set the baseline power for the event channel before it jumps to an event in the library. So here we have one way to set the baseline powers quickly: save a library of events that record the baseline powers for all the channels you are working with, and stop through it to restore the baseline powers.
When it comes to batch classification, we use existing characteristics files, which were produced by a classification processor, to match intervals with an event library. This comparison does not use the current baseline power values. We use the baseline power only when we calculate the metrics used by event classification. Because the classification processor has already calculated the metrics, we do not need the baseline power to compare the metrics of a recorded interval with the metrics of an interval in the event library. We would need the recorded baseline power, however, if we wanted to translate the metrics back into absolute signal power measurements.
An Event List is a list of exceptional moments in the recorded data. The list takes the form of a text file. Each line of the text file defines a separate event. We pick the event list with the event list Pick button. We move through an event list with the Back, Go, and Step buttons. Each of these provokes a Jump to a new interval. Here is an example event list.
M1300924251.ndf 13.0 3 Transient 3.4 0.995 0.994 0.009 0.136 0.408 0.533 M1300924251.ndf 303.0 3 Hiss 3.4 0.710 0.810 0.644 0.383 0.553 0.699 M1300924251.ndf 402.0 3 Other 3.4 0.513 0.595 0.618 0.473 0.559 0.578 M1300924251.ndf 105.0 4 Rhythm 2.8 0.656 0.226 0.441 0.790 0.324 0.688
Each line containes a separate event. Each event is itself a list of elements. The first element is the name of an archive file. The second is the time from the start of the archive. The third element is a list of channel numbers to which the event applies. In the case of seizure detection, we specify only one channel, but it is conceivable that some socially-interactive events could involve more than one channel, in which case the channel element would be a list of channel numbers enclosed in double quotes. The remaining elements are characteristics. In this case we have an event type followed by the baseline power at the time of the event, and six metrics used by the Event Classifier.
The isolate_events parameter in the configuration directs the Neuroarchiver to set channel_select event channel whenever it displays and event. This isolates the event channel for display. Set this parameter to 0 to see all channels.
Whenever we jump to a new event, the Jump checks to see if the Event Classifier window is open, and if so, the routine makes an effort to set the current basline power for the event channel correctly before the Player applies processing. The Jump checks the event description to see if it matches the format of a Classifier event. If so, the Jump assumes the baseline power is in the standard location for such event descriptions. By setting the baseline power in this way, the Jump makes sure that processing is correct, even though it is being applied to one interval in isolation of the history of the signal before and after.
The Event Classifier works with a Classification Processor to compare the characteristics of a test event to those of a library of reference events that we create by visual inspection of recorded signals. We first presented the Event Classifier to our collaborators in October 2011, when we gave this talk, which you may perfer as an introduction to the program.
The Classifier finds the reference event that is most similar to the test event and uses the type of the reference event as its identification of the test event. Thus if a seizure event is the most similar event in the library, we classify the test event as a seizure. The reference events must be produced by the same classification processor we apply to the test event, so that the characteristics may be compared effectively.
When we press the Classifier button, a new window opens up. An event list stored in a text file defines the library of reference events, and we read this into the Classifier with its Load button. Before we use the classifier, however, we must initialize its internal variables by selecting and applying the same classification processor we used to obtain the reference events.
The reference events appear as points on the map and as lines in the text window. The map plots the events in a space defined by two of their characteristics. The characteristics calculated by a classification processor vary from 0.0 to 1.0, and each represent an independent characteristic of the event. We call them metrics. Two menu buttons let us pick the metric we use for the x and y directions in the map. The Classifier obtains these names from the classifier_metrics string, which is initialized by the classification processor. We can change the metrics and re-plot the graphs with the Refresh button. To jump to the event corresponding to one of the points, click on the point. You will see the Player jump to the event, and the event itself will be hilited in the Classifier's text window.
The map shows how well two metrics can distinguish between events of different types. Each event type has its own color code, as set by classifier_types, which is also initialized by the classification processor. We hope to see points of the same color clustering together in the map, and separately from points of different colors. In practice, what we see is overlapping clusters of points, each cluster with its own color.
We can edit and delete events in the text window with the keyboard and mouse. We can add the current interval displayed in the Neuroarchiver with the Add button. We can even type new events in by hand. Each event must have an archive name, a time in seconds, and a channel number. Each event written by the Classifier to the text window has a J button with which we can jump to the event in the Neuroarchiver, and a C button we can use to change the event type. After we edit the list by hand, we press the Refresh button to create a new set of C and J buttons, and plot the new list to the map.
With the Classifier window open, the classification of each playback interval will be displayed in the classification label in the top-left corner, and plotted as a white square on the map. In order for this plotting and classification to occur, interval processing must be enabled, and the processor must produce a characteristics line in the following format.
archivename playtime channelnum lettercode baselinepwr metric1 metric2 ... metricN
The archive name and time come first. If the time is a whole number of seconds, it must have a zero after the decimal point. The first integer is the first channel number. Following this is is a single letter giving an initial clue as to the nature of the event. In ECP1 (Event Classification Processor, Version 1) we use "L" for signal loss, "N" for normal power, and "U" for unknown event. After the letter code comes the baseline power in k-square counts, given to at least one decimal place, as determined by the processor's baseline power calibration. Following the baseline power, there must be at least one real-valued metric after the baseline power.
When we are processing several channels at once, we add the characteristics of subsequent channels to the same characteristics line with further strings channelnum lettercode baselinepwr metric1... metricN. Library events have the same format as the characteristics lines, except the event letter is replaced by an event type word, and reference events refer to only one channel. We note that the reference event list, and the characteristics lines, are all compatible with the format of an event in an event list, so the Neuroarchiver can navigate through the reference library or a characteristics file using its event navigation buttons.
The first metric after the baseline power is the event trigger metric. If its value is greater than classifier_threshold, the Classifier treats the interval as an event. Otherwise it classifies the interval as "Normal". The remaining metrics can represent any feature of the signal, but they must always lie within the range 0 to 1. The ECP1 processor uses a sigmoidal function to restrict the value of its metrics.
Here is an example characteristics line produced for a single channel, No3, by ECP1. There are six metrics after the baseline power. These are event power, transient power, high-frequency power, spikeyness, asymmetry, and intermittency. You can see how the processor obtains these metrics from the signal by looking at the ECP1 script.
M1300924251.ndf 13.0 3 U 3.4 0.995 0.994 0.009 0.136 0.408 0.533
The N real-valued metrics allow us to regard each playback interval as a point in N-dimensional space. Because the metrics lie between zero and one, the intervals are constrained to lie within an N-dimensional unit cube with one corner at the origin. The reference events are likewise constrained to this unit cube. The difference between two events is the N-dimensional distance between them, which is the square root of the sum of squares of the differences between each of their metrics. Thus the Classifier picks the closest reference event to classify a new event, and each new event has a match distance, which is the distance between the new event and the closest event.
When it detects an event, the Classifier finds the closest reference event, hilites this event in the text window, and displays the match distance in the Match label. if the match distance is greater than the match limit, which is set by the Limit entry box, the Classifier gives the current event the type "New". Otherwise it gives the current event the same type as its closest reference event. The purpose of the match limit is to help us identify events that are significantly different from those already present in our library.
To build a reference library, we select our processor, enable it, and start playing an archive. Most likely, the processor uses baseline power calibration, so we may have to let it run for a few minutes to perform this calibration. When we are satisfied that the baseline powers have been well-calibrated, we return to the beginning of the archive and open the Classifier.
In the Classifier, we press Continue. It will play the archive until it detects and event, at which point it will stop. The new event will appear on the map and it will be of type "New". We press Add to add this first event to our library. We examine the signal and set its type with its C button or by hand. The C button will cycle the event type through those listed in the event_types string. When we have the correct event type for our new reference event, we press Continue again. The next event will always be classified as the same as the first, or if its match distance is greater than the match limit, the new event will be classified as "New". We can add the new event, or continue to the next. When we see an event that is classified incorrectly, we add it and correct its type. We proceed until we have a library of events that gives us the correct classification almost all the time.
Once we have a library, we save it with the Save button. This library now allows the Classifier to identify the type of future events automatically. If the metrics are adequate to separate the events of different types, the automatic identification will be effective.
For a discussion of the metrics the ECP1 processor uses to distinguish events, see Similarity of Events. For the exact calculation of its metrics, consult the processor script itself.
The Batch Classifier is an extension of the Event Classifier. We can go through an archive with the Evetn Classifier looking for particular events using the playback and the display in the Classifier window, or we can do so more quickly using Batch Classification. The Batch Classification button opens a new window with its own buttons and check boxes. It applies the reference library to previously-recorded characteristics files produced by the same classification processor. We can select which channels we want to search for events, and which types of events we want to record in our output file. We must select the charateristics files and specify an output file before we start.
The New event type is an event that differs by more than match_limit from all existing events in the library. If we are searching for one particular type of event in our data, such as a Seizure, we could fill out event library with seizure events, and then assume that anything with a match distance of 0.2 or less must be a seizure, and anything else is not a seizure. We set the match_limit to 0.2 in the Classifier window, and now the Batch Classifier will classify each event as wither New or Seizure.
The Batch Classifier makes no use of the baseline power values recorded in the characteristics files it takes as input. The comparison between each interval in the characteristics files and each event in the reference libary is done on the basis of the metrics alone. We need the baseline power to calculate the metrics in the first place, but we do not need the baseline power to compare the metrics.
When we specify the characteristics files, we can select one or more files with the browser. In addition, we can check Consecutive, and the Batch Classifier will look for more characteristics files with the same file name structure as the final file we specified ourselves. Thus if we specified characteristics file M1300920651_ECP1.txt, produced by event classifiaction processor ECP1, the Batch Classifier would first go through the metrics stored in this file, then look for the next file in the same directory with the same structure. In the example above, the Batch Classifier finds file M1300924251_ECP1.txt.
When we specify the output file, in which the selected events are to be stored, we can specify a single file, such as "Events.txt", or we can specify a file with the structure M, ten digits to specify a timestamp, an underscore, and an event file string. For example, we could specify M1300920651_Events.txt. When we specify a file of this structure, the Batch Classifier creates a new event file of the same structure for each characteristics file it reads from disk. Thus we can have an hourly list of events or we can combine all events into one large event list. In any case, each time we apply the batch classification, the output files, if they exist already, are deleted and re-written with the new event lists.
The Event Handler is a program executed by the Event Classifier. When the Classifier and the Player are operating together, we have seen how the Classifier plots the current interval for all selected channels in its map, and classifies all events with its library. At the same time, the Classifier can execute a Tcl script that takes action based upon the nature of the events it encounters during play-back. We call this script the event handler and we enable its execution with the Handler check box. The script itself we store in the handler_script string. If this string is non-empty, the Classifier will attempt to execute it.
Example: We wish to flash a lamp whenever we encounter a Seizure event while playing live data recorded from an animal. We set the handler_script to a program that checks the current event type and executes a sequence of LWDAQ commands if the type is a Seizure. The LWDAQ commands will open a socket to a LWDAQ driver and turn on and off a lamp connected to some LWDAQ device such as the Lamp Controller A2060L.
The Event Handler has access to a selection of Event Classifier local variables, such as type, which contains the type of the current event. The following table lists the variables the event handler can use, and their values.
| Name | Value |
|---|---|
| event | the event itself |
| closest | the closest event to this one in the event library |
| types | the name of the event type |
| fn | the archive file in which the event occurs |
| pt | the play time within the archive at which the event occurs |
| info | the Neuroarchiver_info array |
| config | the Neuroarchiver_config array |
The info and config variables are the Neuroarchiver information and configuration arrays. Thus we would obtain the value of the playback interval with $config(play_interval) and the current recording time with $config(record_end_time). The event variable contains a string describing the current event in the same way it would appear in an event list. The closest variable contains the closest event in the library.
The following example responds to events of type Seizure by writing a message in red to the Neuroarchiver text window, giving the play time and channel number.
if {$type == "Seizure"} {
Neuroarchiver_print "Seizure on channel $id at time $pt." red
}
One way to define the handler script is with a Classification Processor. A Classification Processor already defines the types and colors of events, and the names of the Classifier metrics. It can also define the value of event_handler. The following lines would establish the above handler script for the Classifier.
set info(handler_script) {
if {$type == "Seizure"} {
Neuroarchiver_print "Seizure on channel $id at time $pt." red
}
}
Note that we simply declare the entire script as a string with curly braces marking its beginning and end. Here is another example. We define an event handler that opens a socket to a LWDAQ driver at address 10.0.0.37 and sends hexadecimal command words "0080 DE83 0585 3287 6489 0181" to the device in socket 1.
set info(handler_script) {
if {$type == "Seizure"} {
set sock [LWDAQ_socket_open 10.0.0.37]
LWDAQ_set_driver_mux $sock 1 1
foreach c "0080 DE83 0585 3287 6489 0181" {
LWDAQ_transmit_command_hex $sock $c
}
LWDAQ_socket_close $sock
}
}
If we have an A2060L connected to socket one, these commands will cause it to generate 100 pulses of 5 ms with fixed interval 50 ms and pulse height 10 V.
The Neuroarchiver allows you to inspect the content of recorded data in detail, message by message if necessary. At times the Neuroarchiver might report errors in its text window, something like this:
WARNING: Clock jumps from 43904 to 44060 in M1295029550.ndf at 584 s.
These messages will be in blue. They mean that something has gone wrong in the acquisition of data by the Recorder Instrument. In the example above, the Neuroarchiver detected a jump in the value of the clock message from 43904 to 44060. The next clock message should always be one greater than the last, with the exception of clock message zero, which of course follows clock message 65535. The clock messages are inserted in the message stream by the Data Recorder (such as the A3018) regardless of the incoming transmitter data. They are the messages with channel number zero.
The Data Recorder inserts 128 messages per second, so they are spaced by 7.1825 ms. In the above example, the clock has jumped by 136 instead of 1. We are missing just over 1 s of data. There are several possible explanations for the missing data. One is that the Data Recorder buffer is overflowing because data acquisition is not keeping up with data recording. Another is that data on its way from the Data Recorder to the LWDAQ Driver is being severely corrupted, and the error correction used by the Recorder Instrument is chopping out large chunks of the data in order to make sure it does not pass on corrupted messages.
Most corruption of data from the Data Recorder to the LWDAQ Driver occurs because of extraordinary electrical events like static discharge. In these cases, a few extra bytes are inserted into the data stream by spurious pulses on the logic lines. Starting with LWDAQ 7.5, the Recorder Instrument provides error-correction so thorough that it will almost always be able to remove the spurious bytes and restore all but one or two of the original messages. Thus a warning like the one above will be unusual. Instead, we expect to see clock jumps of at most one or two steps.
The Neuroarchiver lets you look more closely at the incoming messages, which is useful when diagnosing problems. Try clicking the verbose check box. Now you will see more detailed reports of reconstruction in the text window. Press Configure and set show_messages to 1. Press Step in the Player. You will see detail of the number of errors in your playback interval, and a list of the actual message contents, as provided by the print instruction of the Recorder Instrument's message analysis. If there is an error in the playback interval, the list of messages will center itself upon that error. Otherwise the list will begin at the start of the interval. You set the number of messages the Neuroarchiver will print out for you with the show_num parameter.
Here we list changes in recent versions that will be most noticable to the user. You will find the source code here.