The Neuroarchiver Tool is a program written in TclTk. It is a tool available in the LWDAQ Software. It is works with the LWDAQ Software's Recorder Instrument and data acquisition hardware such as the Data Receiver (A3018) and collections of Subcutaneous Transmitters (A3013). The Neuroarchiver acquires raw data continuously from the hardware and saves the data to disk. Independently from its data-storage activities, the Neuroarchiver reads raw data from disk, extracts signals from the raw data, displays the signals on the screen, and applies user-defined analysis to the signals.
The Neuroarchiver makes heavy use of disk files. The recorder portion of the Neuroarchiver acquires data and stores it to disk immediately, without any analysis. The player portion of the Neuroarchiver reads data from disk and analyses it. Both the recorder and player can run simultaneously, so that the player can be reading data as it becomes available on disk.
To run the Neuroarchiver, download and install the LWDAQ Software for your Linux, Windows, or MacOS computer. Run the program and select the Neuroarchiver from the Tool menu. Before you proceed, however, we recommend you open the Recorder Instrument in the Instrument menu, and learn to use it with the help of the Recorder Instrument Manual. Before you acquire data with the Neuroarchiver from your own hardware, you must set up the Recorder Instrument to be the source of data. You are welcome to use the Recorder Instrument's default settings, which will acquire raw data from our demonstration hardware.
The Configure button opens an array of configuration parameters. The Help button offers some basic help. All the Browse buttons allow you to select the directory or file in a nearby text entry box.
In the figure above, the recorder portion of the Neuroarchiver is downloading data in half-second intervals and storing them to an archive file whose name appears to the right of the Archive label. Every 3600 seconds the recorder will produce a new archive file. The archive files are in the NDF (Neuroscience Data Format) we describe here.
Meanwhile, the player portion of the Neuroarchiver is reading from a different archive, which contains data from four Subcutaneous Transmitters (A3013A) implanted in live rats. These four transmitters have channel numbers 1, 2, 6, and 14. The signal versus time is in the left plot and the aplitude versus frequency is in the right plot. The player state is Play with a green background. The green background means that the player is at this very moment analyzing its most recently-read block of messages. If the background is yellow, that means the player is waiting from data to be added to its archive. The player will be waiting regularly if it is playing the same file to which the recorder is writing new data. When the yellow background appears behind the recorder state, the recorder is waiting for data from the Recorder Instrument, which is in turn waiting for data from the data acquisition hardware.
The player displays channels seclected by the channel select string. In our example, the channel select string is "*", which selects all active channels in the time interval being displayed. We discuss channel selection in more detail below. All active channels are listed in the Channel Activity string, which gives the channel numbers and the number of messages received from each channel during the playback interval.
Here is how you start recording and simultaneous play-back with the Neuroarchiver. The Neuroarchiver uses the Recorder Instrument to acquire new data, so the first steps are to set up the Recorder Instrument to point to your Data Receiver (such as the A3018).
Look at your data recorder. The EMPTY light should be flashing regularly. If it is not flashing, your Neuroarchiver is not acquiring data as fast as the data is being recording. This failure to keep up with the pace of recording may be due to the fact that your computer is spending too much time plotting graphs and calculating transforms. Turn off the player's transform display and press Reset again. If the EMPTY light still does not flash regularly, try turning off the signal display, and then stopping playback alltogether. If the red light still does not flash, your problem may be that the network connection is too slow to transport the data as it is created.
Once you get the recording and play-back working, you can try out various values of recording interval and play-back interval. For the most stable operation with up-to date signal display, the recording interval should be half the play-back interval. In stable operation, the player is waiting for the recorder to save data to disk. When the data is available, the player displays it. While it waits, the player state indicator is yellow.
You can look through previously-recorded archives even while you are recording a new archive. Stop the simultaneous playback and select a new archive. If you want to see an overview of an entire archive, select it in the player and press the Overview button.
The two plots are drawn during playback of recorded data. Each plot has its own Enable check-box. Plotting can slow down play-back. You can disable the plots by un-checking the box.
The Value versus Time graph, which is on the left of the Neuroarchiver panel, shows signal voltage variation during the most recent playback interval. In the example display we see four traces. This plot is identical to the one generated by the Recorder Instrument. The traces are color-coded. For a table of channel numbers and their colors, see the Message Display section of the Recorder Instrument Manual, or this picture. The vertical axis is voltage, measured in ADC counts. Each transmitter converts its analog input into a sixteen-bit value. Sixteen-bit values run from 0 to 65535. In the example, the plot spans the range 0 to 65535. The bottom of the range is zero because the value of v_offset is 0. The top of the range is 65535 counts above the bottom because v_range is 65535. To convert between ADC counts and voltage at the transmitter input, consult the transmitter manual.
Example: The A3013A transmitter manual has a section called Analog Inputs. Here you will find that the gain of the transmitter from analog input to the ADC is 300. The voltage range 0 to 65535 corresponds to voltages 0 V to VBAT at the ADC input. For most of a transmitter's operating life, VBAT is 2.7 V, so each ADC count is 140 nV at the X input. The amplifier AC-couples the X input, placing its average value at 1.8 V. The dynamic range for signals at X is −6 mV to +3 mV. We can deduce the battery voltage, VBAT by looking at the average value of the signal in the plot. If A is the average value of X as a fraction of 65535, we have VBAT = 1.8 V / A. From the signal present in archive M1259065886.ndf, we estimate A is 0.64 for channels No1 and No2, 0.70 for channel No14, and 0.8 for channel No6. This implies VBAT of 2.8 V for No1 and No2, 2.6 V for No14, and only 2.2 V for No6. You can compare these voltages to this graph to estimate how much more operating life each transmitter has left.
The horizontal axis in the Value Versus Time graph is time. The full range from left to right covers the most recent playback interval. This interval is shown in the Interval selector beneath the graphs.
We can obtain a close-up of fluctuations in the analog signal by entering AC for v_coupling and changing v_range to a smaller value. With AC coupling, the Neuroarchiver places the average value of the signal half-way up the display.
The right-hand graph is the fourier transform of the signals on the left-hand graph. The frequency range is from f_min to f_max. Each division along the horizontal is one tenth of the range. The amplitude range is zero to a_range in ADC counts. With f_step you dictate the frequency resolution of the transform.
Example: In the example plot, the amplitude range is 100 counts. Each vertical division is 10 counts. These example traces were recorded in London, where the mains frequency is 50 Hz. The amplitude of the transform at 50 Hz is no higher than for neighboring frequencies, at around 400 counts. These signals are recorded from the A3013A, for which one count is 140 nV, so the 50 Hz content is less than 60 μV.
Smaller frequency step means longer computation time for the transform, but a more reliable representation of the signal's frequency content.
The Neuroarchiver works with NDF (Neuroscience Data Format) files. An NDF file contains a header, metadata string, and a data block to which we can append at any time without writing to the header or the data string (although the Neuroarchiver almost always writes to the metadata string whenever it adds data to a file). We define and describe the NDF image format in the Image Format section of the LWDAQ Manual. The Neuroarchiver manipulates these with a library of NDF-handling routines provided by LWDAQ. We list these routines below. These routines are declared in LWDAQ's Utils.tcl script, and you will find them described in the LWDAQ Command Reference. Their names begin with LWDAQ_ndf.
The Neuroarchiver works within a single directory, as named in the Directory entry box on the top of the Neuroarchiver Panel. All NDF archives, analyzer scripts, and results files must be contained in this same directory. If you attempt to use or specify a file outside the Neuroarchiver's directory, it will refuse and print a red error in its text window.
The Neuroarchiver's player will roll over from one file to the next as the recorder creates a new file automatically. Each data file is named with in the form "Mxxxxxxxxxx.ndf", where the ten x's are an integer giving the number of seconds since 1900 hours on 31st December, 1969. This number is the standard UNIX timestamp. From the name of each file, we can determine the time, to within a second, at which its first clock message occured. From there we can count clock messages and determine the time at which any other part of the data occured.
The data processed by the Neuroarchiver takes the form of a list of data recorder messages, as we describe here. In general, the data will contain values from one or more channels. The Neuroarchiver selects which channels to display, transform, and store to disk using its channel_select parameter.
In its simplest form, channel_select is a single "*" character. With channel_select set to "*", the Neuroarchiver looks through the playback interval data and counts how many messages it contains from each of the 16 possible subcutaneous transmitter channels: No0 through No15. Of these, No0 is the clock message channel and No15 is the slow data channel. Transmitters like the A3013A use the fast data channels, No1 through No14. If a channel contains more than activity_threshold messages, the Neuroarchiver considers it active. It plots all active channels and lists them in the Channel Activity string in the format id:qty, where id is the channel number and qty is the number of messages. You will find the activity_threshold parameter in the configuration array (press Configure).
You can select particular channels with a specific channel_select string. You could enter "1 2 6 14" and the Neuroarchiver will attempt to display these channels, even if they have very few messages. You can specify the nominal sampling frequency and scatter extent for each channel. For a description of sampling frequency and scatter extent, see Analysis section of the Recorder Instrument Manual. If you list the channel numbers on their own, the Neuroarchiver assumes a the frequency and scatter given by default_frequency and default_scatter. You specify channel number, frequency, and scatter with three numbers in the form c:f:s. Thus "5:512:8" means channel 5 with sampling frequency 512 SPS and scatter extent 8.
The sampling frequency and scatter extent are used by the Neuroarchiver when it reconstructs an incoming message stream. The Neuroarchiver uses its clocks_per_second and ticks_per_clock parameter to convert samples per second into a sample period in units of data recorder clock ticks. The Neuroarchiver can then go through a channel's messages and identify places where messages are missing, and eliminate bad messages that occur in the message stream at random times.
Reconstruction of a message stream is computationally intensive. The Neuroarchiver applies reconstruction to the selected channels only if the percentage of messages lost from the stream exceeds the loss_threshold, which is a percentage of the expected number of messages, or if the number of extra messages exceets suprious threshold, which is an absolute number of messages.
The loss_threshold parameter has its own entry box in the Neuroarchiver window. With the threshold set to 1%, any channel missing more than 1% of its messages will be reconstructed. Look at the example display, and you will see reconstruction applied to those channels with loss greater than the loss threshold. Reconstructed channels contain the nominal number of samples, which the Neuroarchiver determines from the sample rate (default 512 SPS) and the playback interval.
The spurious_threshold parameter also has its own entry box. Channels can contain more messages than they are supposed to if there is a lot of interference around, or if the signal from the transmitter is weak compared to the ambient interference. The extra messages are bad messages. We eliminate most bad messages with reconstruction.
The Neuroarchiver uses the Recorder Instrument to obtain live data. First you set up the Recorder Instrument to read out streams of messages from a data-recording device, then you open the Neuroarchiver to store the live data. The Neuroarchiver does not display new data until after it has been stored to disk, so display and analysis are not part of data acquisition.
To capture live data, open the Recorder Instrument and configure it to read your data out of your data recorder. The Neuroarchiver will simply call the Recorder Instrument's data acquisition procedure when it captures new live data. You don't have to leave the Recorder Panel open after you set it up for data Recording, but you can leave it open if you like. By default, the Neuroarchiver turns off the Recorder Instrument's plotting and analysis of the incoming data. But you can turn on the plotting and analysis by setting enable_recorder_analysis to 1 in the Neuroarchiver's configuration panel, which you can open with the Configure button. With the plots turned off, you will see the raw data displayed as gray-scale pixels in the Recorder Instrument.
The only thing that passes from the Recorder Instrument to the Neuroarchiver is the raw data acquired from the data acquisition hardware. The Recorder Instrument has a data acquisition parameter called daq_num_clocks. When you instruct the Recorder Instrument to acquire new data, it acquires a block of messages with exactly this number of clocks. The Recorder Instrument makes sure that first message in the block is always a clock message. The Neuroarchiver calculates daq_num_clocks from record_interval, which has units of seconds. In the example shown above, the recording time interval is 0.5 s, and is shown in the menu-button to the right of the recorder controls.
As the recorder saves data to a file, it also adds timestamp fields to the file's metadata string. We discuss timestamps and metadata below. You can view the recording file's metadata with the recorer's Metadata button.
The NDF format contains a header, a metadata string, and a data block. Transmitter messages and clock messages are stored by the recorder in the data block. New data is appended to the data block without any alteration of existing data. The metadata string has a fixed space allocated to it in the file, but is itself of variable length, being null-terminated. The Neuroarchiver makes use of the metadata string in three ways.
When the recorder creates a new file, the metadata contains two comment fields. These fields are delimited by xml "c" tags.
<c>Date Created: Wed Nov 25 21:23:21 2009.</c> <c>Creator: Neuroarchiver 37, LWDAQ_7.3.8.</c>
You can view, edit, and save the metadata for the recording file with the Metadata button in the recorder section of the Neuroarchiver. You can do the same for the metadata in the playback file with the player section Metadata button. We ask users to enter their own comments in fields delimited by "c" tags.
A timestamp field contains two numbers and is delimited by "t" tags. The first number is time in seconds. The second number is the index of the clock message that occured at that time after the first clock message in the file. The timestamp "0 2" means the first clock message is message number 2 in the data block. Most often, the first timestamp is "0 0". The timestamp "2 4293" means the clock message that occurs 2 s after the first is message number 4293.
You can see how timestamp fields can accelerate navigation through a data file. In the player, we can jump to a new location in a file by entering a value in the Time box and pressing Step. The Neuroarchiver uses timestamps to get close to the requested time. Timestamps make the Overview button possible. Without timestamps, the Neuroarchiver does not bother to try to create an overview of an entire data file.
Files created by Neuroarchiver Version 35 and earlier contained no timestamps. You can generate new timestamps for a file with the New Timestamps button in any metadata viewing window. Generating new timestamps takes roughly a minute for every hour of data recorded.
The recorder stores a new timestamp to the recording file every time it writs a new block of data to disk. After a while, the metadata string might fill up. When this happens, the recorder condenses the timestamp list by deleting every other timestamp, and then adds its new timestamp before writing the new metadata to disk. In this process, it alters only the timestamp fields in the metadata string. Any user-created comments will remain intact.
The Neuroarchiver performs all its signal processing, plotting, and analysis when it reads data from disk. Reconstruction of the signal occurs when the signal loss exceeds the Reconstruct Loss Threshold. You turn on the transform calculation with the Transform check box. Both plots have their own enabling check boxes.
If you check the Verbose box on the Neuroarchiver, the player will report on its reading and analysis of data. You will see the loss in each channel, and the results of reconstruction and extraction of message from the playback interval's data. In the example display, you can see the player telling us that it extracted two channels and reconstructed the other two.
You can navigate through archives by entering a new Time value and pressing Step. You will see the end time of the archive, which is the archive length in terms of the number of clock messages it contains, on the far right of the player section. If the archive has timestamps in its metadata, moving through the file will be fast. Otherwise, movement is slow. Try pressing Metadata and then New Timestsamps. Wait until you see the new timestamps appear and press Save. Now movement will be faster. The Back button steps back one playback interval.
The Overview button opens the player archive and creates a condensed view of its entire contents. The overview opens in a separate window and takes a few seconds to generate. The Neuroarchiver goes through the list of timestamp fields in the file's metadata and reads a small segment of data from the file at each moment specified in the timestamps. It finds the messages for each channel in this segment and adds them to a separate graph for each channel.
The overview plot will follow the settings given in the player for the Value vs. Time plot. It will include all channels that contain at least activity_threshold messages in the entire data block. You can leave the overview windows open and select a new playback archive for a new overview. Thus you can compare overviews on the screen.
Following the playback analysis, which may include reconstruction and fourier transform, the Neuroarchiver will apply a user-written analyzer script to the contents of each selected. Select the analyser script with its Browse button. Results of analysis will be appended to the results_file, whose name is Results.txt by default, but you can change in in the configuration panel (press Configure). Here is an example analyzer script.
#
# Start.
#
# This script will be called by the Neuroarchiver once for each
# channel in the channel list. Each time, the channel number is
# stored in a variable called "id". The channel signal is stored
# in "info(signal)". Each point in the signal is represented by
# a timestame and a value, so that the info(signal) string is
# a sequence of alternating timestamps and values separated by
# spaces. The values are unsigned sixteen-bit integers. The
# timestamps are unsigned twenty-four bit integers.
#
#
# The "result" string is empty when we analyze the first channel
# in the "channels" list. We put the name of the archive and the
# play index in the string before we analyze the first channel.
#
if {$result == ""} {
set result "$config(play_file) $config(play_time) $num_clocks "
}
#
# The Neuroarchiver_signal_summary routine returns the average,
# standard deviation, maximum, and minimum values of the current
# signal. The following line calls the routine and scans its
# result string for the four parameters.
#
scan [Neuroarchiver_signal_summary] %f%f%f%f ave stdev max min
#
# We format the standard deviation and append it to the result
# string along with the channel id and the number of messages
# we received from the channel.
#
append result "$id [format %.1f $stdev] $info(num_received) "
#
# End
#
The script is applied by the Neuroarchiver to each of the selected channels. Thus it contains instructions that apply to the signal and spectrum of the current channel. It also contains instructions that are to be executed before analysis of the first channel. We detect the first channel by looking at $result, which will be empty before we analyze the first channel, but non-empty afterwards, as the analyzer script appends values to the string.
Because the script is TclTk, it can do just about anything that TclTk can do. In theory, it can e-mail the finished result string to you, or upload it over the network to a server. In this example, it constructs a single-line result string, and allows the Neuroarchiver to write the string to the result_file.
The $info(num_received) value is the number of messages received in this playback interval. The $num_clocks value is the number of clocks in the playback interval. The script has access to all the Neuroarchiver configuration parameters with the $config(paramname) construction.
The script has access to the extracted or reconstructed signal as $info(signal). The signal takes the form of a sequence of numbers separated by spaces. Each pair of numbers is the time and value of the signal. The time is in clock ticks from the start of the playback interval. The value is in sixteen-bit ADC counts.
The script has access to the fourier transform of the signal, assuming it has been calculated, as $info(spectrum). The spectrum is a sequence of numbers separated by spaces. Each pair is a frequency and an amplitude. The frequency is in Hertz and the amplitude is in sixteen-bit ADC counts.
The Neuroarchiver Script is the TCL/TK program that defines the Neuroarchiver process. The name of the file is Neuroarchiver.tcl, and you will find it in the Tools directory of your LWDAQ distribution. We have added extensive comments to the script. We intend these comments to act as an introduction to TCLTK and LWDAQ programming.