Recorder Instrument

© 2007-2017 Kevan Hashemi Open Source Instruments Inc.

Contents

Description
Recorder Messages
Data Acquisition
Recorder Script
Message Display
Channel Selection
Message Analysis
Error Correction
Data Processing

Description

Note: This manual applies to the Recorder Instrument bundled with LWDAQ 8.4.9+.

The Recorder Instrument is a computer program written in TCL/TK. It is one of the Instruments available in the LWDAQ Software. The Recorder Instrument reads messages from LWDAQ (Long-Wire Data Acquisition) devices such as the Octal Data Receiver (A3027), Data Receiver (A3018), or Animal Location Tracker (A3032). The first two are data recorders. The last one is a location tracker. They present the LWDAQ with a message buffer and a reset command. The message buffer is a byte-by-byte, first-in first-out buffer (FIFO). The messages it contains are four-byte records with an additional payload of specialized data. The four leading bytes contain an eight-bit channel number, a sixteen-bit data word, and an eight-bit timestamp. The A3018 and A3027 record messages with no payload. Their message buffers are 512 KB long, so they can hold up to 131k messages. The A3032 records messages with a sixteen-byte payload, so its 512 KB buffer can hold 26k messages. The reset command clears the message buffer and sets the data recorder's clock to zero. An animal location tracker resets its message buffer, but it receives its clock messages from a separate data receiver, so its clock will not be reset.


Figure: Recorder Instrument Panel on MacOS. On Linux and Windows, the Recorder will appear with different style buttons and colors, but will have the same layout and functionality. Here we have six 1024 SPS channels and two 512 SPS channels. All are picking up 60-Hz mains hum to varying degrees, except No3, which is responding to a 5-mV frequency sweep. Although there is no evidence in the recording, the eight channels come from four dual-channel transmitters, each operating with a different set number.

When you open the Recorder Instrument in the LWDAQ Software, it looks like the figure below. You can download the software here. Run the program and select Recorder from the Instrument Menu. Press Read to read in some data from disk. Select one of the recorder files in LWDAQ's Image directory.

The Reset and Configure button resets the data recorder's data buffers and message clock. When you execute the reset, the EMPTY indicator on your data recorder flashes briefly. The data recorder is always storing its own clock messages, so the EMPTY light won't stay lit for long. The same button press configures the data recorder to record from a particular set number, as determined by the set_num parameter. The set number is the top four bits of a signal's channel number. By default, all data recorders are configured for set zero (0), which contains channel numbers 1-14. Enter a wildcard "*" in set_num to configure the data recorder to combine signals from all sets, and it will record channel numbers 1-223.

The Recorder Instrument acquires data in discrete blocks. It assembles these blocks in memory until it can compose an interval with exactly daq_num_clocks clock messages. The fixed number of clock messages implies a fixed length of time for each acquired interval of data. As soon as the Recorder Instrument has a complete interval, it reads and displays the firmware version of the data recorder. The firmware version and the assembly number (A3018, A3027, A3032, etc.) together allow us to deduce the features suported by our data recorder.

Example: The Data Recorder (A3018A) produces 128 clock messages per second, so when daq_num_clocks = 128, each interval of messages acquired by the Recorder Instrument spans one second of data recording.

Note: Each time it acquires a block of data, it opens and closes a TCPIP socket to the LWDAQ Driver. Each time it closes its socket, the LWDAQ Driver is available for another client. As a consequence, several Recorder Instruments running in several separate instances of the LWDAQ software can download data from separate data recorders attached to the same LWDAQ Driver.

The Acquire button obtains one interval of data from the data recorder. The Loop button obtains repeated intervals until you press Stop or Acquire. The Activity line lists of all active channels, giving channel numbers and the number of messages in the current interval for each channel. Channel zero should always be present, with a number of messages equal to daq_num_clocks. Other channels will be listed so long as they contain more than active_threshold messages. The active_threshold parameter is available in the Info panel.

You can save the messages the Recorder Instrument acquires from the data recorder with the Write button, and read back saved data with the Read button. The LWDAQ software supports three formats for recording data to disk: GIF, DAQ, and NDF, as we discuss here. Each time you press the Write button, you create a new file on disk or replace an old one. If you want to append new data to an existing file, consider using the Neuroarchiver, which appends dadta to NDF files.

With analysis_enable = 1, the Recorder Instrument analyses each interval it acquires. It plots the data and generates a result string. You can see an example of a result string in the screen shot above. To understand the meaning of numbers in theresult string, set verbose_result to 1, and acquire or read some data. You will get something like this:

Recorder_1
Channel Number: 1
Number of Samples Recorded: 512
Average Value: 43758.6
Standard Deviation: 117.9

The data recorder receives messages from up to 255 channels. Our subcutaneous transmitters use channel numbers 1-223 for their signals, with the exception of those that have remainder 15 when divided by sixteen. These are are reserved for auxilliary data. You determine which channels the Recorder Instrument displays with the analysis_channels string. By default, this string is "*", which selects all channels for which there are at least activity_threshold messages in the interval. Each channel receives four numbers in the result string. The first is the channel number, the second is the number of messages received from this channel, the third is the average value of the message samples, and the third is the standard deviation of the sample values. The numbers provided for the clock messages are slightly different. Instead of the average and standard deviation, the Recorder Instrument provides the minimum and maximum data values.

Example: The Subcutaneous Transmitter (A3013A) provides one analog input at 512 SPS. The gain in the analog input amplifiers is ×300, and the input range of its 16-bit ADC is 0 V to 2.7 V. Each ADC count is 2.7 / 65536 / 300 = 137 nV. The example Recorder Instrument result given above is from such a transmitter with its input wires open-circuit. The standard deviation of 117.9 implies open-circuit noise of 16 μV. If we connect the wires together with a 10-MΩ resistor, the standard deviation drops to 90, or 12 μV.

When downloading from a location tracker, each message will have a payload of power measurements. Any time the messages come with a payload, set the payload_length parameter to the number of bytes in the payload. The total size of the message will be four bytes plus the payload bytes.

Recorder Messages

The messages in the data recorder's message buffer are each four bytes long followed by a payload of length set by the data recorder of location tracker. The bytes of each message are listed in the table below. The Channel Number is used to identify the source of the message. Channel number zero is reserved for clock messages. In the case of Subcutaneous Transmitters, the channel number will be in the range 1-223. Transmitters may belong to different sets, but the messages acquired and stored by the Recorder Instrument do not contain any information about the set number. Only the data recorder is aware of the set numbers of incoming messages, and it removes the set number from the message before storing the message in its memory.

Following the channel number, each message contains a sixteen-bit data word. In the case of SCTs, the sixteen-bit data word is a digitized voltage. The last byte of the message is a timestamp.

ByteContents
0Channel Number
1Most Significant Data Byte
2Least Significant Data Byte
3Timestamp or Version Number
4 to (L+3)Payload Bytes
Table: Contents of One Message. Each message consists of four bytes followed by a payload. Here the number of payload bytes is L.

All data recorders have a clock. The clock is a twenty-four bit counter that counts clock ticks. In the Data Receiver (A3018), the frequency of the clock ticks is 32.768 kHz, provided by a precision clock oscillator. The bottom eight bits of the counter provide the data recorder with the timestamp value it places at the end of each message (with the exception of clock messages). Every time the timestamp wraps around to zero again, which is every 256 ticks of the clock, the data recorder stores a clock message in its message buffer. The clock message has channel number zero. Instead of a timestamp, the data recorder stores a version number for use by the Recorder Instrument to detect incompatibility between software and hardware. The data in the clock message is the upper sixteen bits of the clock counter. The upper sixteen bits increment every time the data recorder stores a clock message.

In the case of the 32.768 kHz reference clock, we get a clock message 128 times a second, or every 7.8125 ms, and the sixteen-bit clock message value wraps around to zero again every 512 s. We call this 512-s wrap-around period the timing cycle. Meanwhile, clock ticks occur every 30.5 μs. By looking at clock messages and message timestamps, the Recorder Instrument can figure out where in the timing cycle every message took place, and do so with 30-μs precision.

Example: Our data recorder is an Octal Data Receiver (A3027). Its reference clock is 32.768 kHz. It stores clock messages with channel number zero 128 times a second. Within our laboratory are ten animals with implanted Subcutaneous Transmitters (A3028). The transmitters transmit at various sample rates between 64 SPS and 2048 SPS. Each block of messages we read out contains dozens of messages from the various transmitters between each two clock messages. Each transmitter message has a timestamp between 0 and 255. This timestamp gives us the time at which the data recorder received the transmitter message, in reference clock ticks since the previous clock message.

Example: Our data recorder is the hypothetical Data Logger. Its reference clock is 16 Hz. It stores clock messages with channel number zero every sixteen seconds. Within a factory hall are one hundred wireless thermometers, each with a unique channel number. These transmit a temperature measurement whenever their measured temperature deviates by 0.1°C from the previously-transmitted temperature. In the morning, when the factory lights turn on, the thermometers transmit every few seconds. In the mid-morning, they transmit every few minutes. When the factory lights turn off, they transmit every few seconds again. During the day, a computer monitors the temperatures. If a part of the factory gets to hot or too cold, the computer issues a warning. The thermometers themselves are similar in design to the Subcutaneous Transmitters (A3013), except their operating life is three years instead of two months.

The data recorder will never store a message with channel number zero unless that message comes from the clock. All messages with channel number zero are guaranteed to be clocks.

The Recorder Instrument supports the use of auxiliary channels, which transmitting devices can use for acknowledgements or slow data. All channel numbers which, when divided by sixteen, have remainder fifteen, are auxiliary channel numbers. They are channel numbers 15, 31, 47, 63, 79, 95, 111, 127, 143, 159, 175, 191, 207, and 223. The aux_list_name parameter is the name of the string that contains a list of auxiliary messages. This list will include only the most recent aux_num_keep messages. Other processes waiting for data or acknowledgement can search the auxiliary message list for the messages they need. When they find such a message, they delete it from the list.

Each auxiliary message contains sixteen bits of data. The Recorder Instrument assumes these sixteen bits are arranged as a four-bit source identifier, a four-bit field address, and an eight-bit data byte. The four-bit source identifiers we combine with the top four bits of the auxiliary channel number to obtain the channel number of the transmitter that produced the auxiliary message. The four-bit field addresses 0 and 15 are reserved. The others may be used to identify instructions, registers, or a device group. The data has no constraints upon its value, and can be used for any purpose. Each entry in the auxiliary message list contains four unsigned integers: channel number, field address, data, and timestamp. The timestamp is the Data Recorder time at which the auxiliary message was received. Its units are clock ticks. The timestamp allows other processes to use the precise timing of auxiliary messages as a form of data addressing. The Recorder Instrument provides the timestamp modulo 216, which is sufficient for any timestamp-based addressing between two clocks running within ±20 ppm of one another.

Example: The Implantable Sensor with Lamp (A3030C) with channel numbers 1-14 uses channel 15 to send command acknowledgments. The identifier bits are the lower four bits of the ISL channel number, while the upper four bits of the channel number are zero. The field address is one (1) to indicate an acknowledgement. The data byte is a value passed to the ISL by the acknowledgement request. Once the ISL Controller has detected the acknowledgement, it deletes the acknowledgement message from the auxiliary message list.

The following table gives the existing definitions of field address values and the meaning of the data that follows them.

Field
Address
Field
Function
Data
Function
0ReservedUndefined
1AcknowledgementAcknowledgement Key
2SynchronizeDevice Timestamp
3StatusFlags for Device Status
4Time-Addressed DataData Byte
5..14UndefinedUndefined
15ReservedUndefined
Table: Defined Field Addresse and Data Functions.

The time-addressed data is data transmitted with a particular timing pattern that allows the receiving process to figure out which byte is which. Such a scheme might be as simple as transmitting the high and low byte of a sixteen-bit temperature measurement on consecutive clock ticks, or it might be as complex as a pseudo-random sequence of time delays that carries thousands of unique bytes of data over a period of several minutes. In all cases, the intent is to allow slow data to pass from transmitter to receiver without interrupting the continuous flow of faster data, and in such a way as to tolerate loss of some of the data. Thus we use timing as a way to detect the loss of a message, which allows the receiver to request that missing bytes be transmitted again.

Data Acquisition

To acquire fresh data from a data recorder, you must set up a LWDAQ (Long-Wire Data Acquisition) system with a data recorder, and make it accessible to your computer. We describe how to set up an LWDAQ in the LWDAQ User Manual.

Example: Press Acquire in the Recorder Instrument immediately after launching LWDAQ. The chances are good that the Recorder Instrument's default settings will point to a LWDAQ and data recorder we have left on the Internet for your benefit, and you will get some live data. Please refrain from taking data for more than a few minutes, as others are likely to need to use the same hardware.

Once you have a LWDAQ available, consult the manual of your data recorder for information on how to connect it to the LWDAQ. Chances are, all you have to do is plug it into one of the LWDAQ Driver sockets with a CAT-5 patch cable.

The Recorder Instrument provides a set of lebeled text entry boxes on the right side of its window. Among these, you will see several that begin with daq_. We describe each of these in the table below. You change the value of the named parameters by entering new values in the text entry boxes.

ParameterFunction
daq_ip_addrIP address of LWDAQ driver
daq_driver_socketdriver socket number used for data recorder
daq_mux_socketmultiplexer socket used for data recorder (if any)
daq_num_clocksnumber of clock messages to accumulate per acquisition
payload_lengthnumber of payload bytes following message core
set_numthe transmitter set number the data recorder should detect
Table: Recorder Instrument Data Acquisition Parameters. You are most likely connecting your data recorder directly to the driver with a cable, so you won't have to bother with daq_mux_socket at all.

When you press the Acquire button, the Recorder Instrument obtains a new interval of messages from the data recorder. In LWDAQ terminology, the Recorder Instrument acquires an image. This image contains exactly the number of clock messages you specify in daq_num_clocks. If you specify n, the first message in the image is clock message 0 the last message is the one that preceeds the n+1'th clock message. The next image the Recorder Instrument acquires will begin with this same n+1'th clock message, but in the next image this clock message will be clock message 0. The time it takes the data recorder to generate daq_num_clocks clock messages is the image time. The image contains all messages stored by the data recorder during the image time.

In order to assemble images with a specified number of clock messages, the Recorder Instrument reads blocks of messages out of the data recorder and adds them to its own message buffer. When the message buffer contains enough clock messages, the Recorder Instrument extracts the desired interval from the message buffer and stores these messages in the new image. It leaves unused messages in the message buffer for inclusion in the next image. No messages are lost unless you leave the data recorder too long and it's message buffer overflows, in which case the oldest messages are lost because they are over-written by new messages. But the Recorder Instrument does not throw any data away unless you press the Reset button. When you press the Reset button, the Recorder Instrument clears its message buffer and clears the data recorder message buffer as well.

Example: We have a Data Receiver (A3018) connected to our LWDAQ. Its Loop Antenna (A3015A) is receiving signals from four Subcutaneous Transmitters (A3013A). Each of the A3013As transmits 512 messages per second. The channel number of their messages is equal to their Transmitter Identification Number. The Data Receiver stores 128 clock messages per second. When we set daq_num_clocks to 128, the image period is one second. In this case, the image contains 128 clock messages plus 2048 transmitter messages. The combined size of this interval is 8704 bytes. We turn off all the transmitters and acquire again. We see the transmitter traces. We acquire again. They are still there. We are reading data that is several seconds, or even several minutes, old. We press the Recorder Reset button and acquire again. Now we see only a red trace. This is the clock message trace. It has a slight slope to it.

Suppose your data recorder is recording messages faster than the Recorder Instrument can read them out. Eventually, you will start to lose messages. The data recorder's message buffer will overflow. When the buffer overflows, the data recorder will over-write the oldest messages with the newest messages. The Recorder Instrument will recognise that the latest block of messages it downloads from the data recorder does not match up with its message buffer. When the Recorder Instrument detects this or any other kind of error in the data it receives, it resets the data recorder's buffer and message clock.

Example: We turn on our four transmitters. Our Data Receiver (A3018) stores 8704 bytes per second. But our download speed from the data recorder to the Recorder Instrument over our local network is 4000 bytes per second. After a hundred seconds, the Data Receiver's 512 KB buffer is full of messages waiting to be downloaded. The buffer overflows. The Data Receiver detects a jump in the clock messages of the interval it is trying to compose, discards the data it has not yet analyzed, and resets the data recorder's buffer and clock.

Set analysis_enable to zero and acquire again. The message traces disappear. All you see is a black image with some of the top rows filled in with random pixels. These pixels are your messages. They usually occupy only a small fraction of the display area, although they can overflow the image area when we are recording from a location tracker. Press the Info button. You will see another window with many more Recorder Instrument parameters (screen shot). Among them are daq_image_height and daq_image_width. These determine the size of the image that holds our messages. By default they are 300 and 400. The first row of the image we use for a header, so that leaves 119,600 bytes for messages, which is space for 29,000 messages.

When you save a Recorder Instrument image to disk with the Write button, we recommend you give it the extension .daq. That way, LWDAQ will store the image in its DAQ format. The black part of the image following the actual messages will be left off the file. Only the header and the actual messages will be stored. But the Neuroarchiver, uses the NDF format to store messages. As it acquires new messages, it appends them to the data section of an existing NDF file.

Example: With four transmitters active, each transmitting 512 messages per second, we we obtain 2048 transmitter messages and 128 clock messages, making a total of 2176 messages of four bytes each, or 8704 bytes. The A3018's message buffer is 512 KBytes, so it will hold up to one minute of data for us. Because of this ample buffer in the data recorder, our instantaneous TCP/IP download speed from the LWDAQ driver can drop to zero for ten or twenty seconds, and we will not lose any data. So long as our average download rate is around 8 Kbytes/s (64 kbits/s) or greater, we will sustain the data acquisition without loss. We use the Neuroarchiver to store the dta to disk, we will accumulate roughly 8 Kbytes/s, or 30 Mbytes/hr.

When you read Recorder Instrument images from disk, the Recorder Instrument displays them as if they were freshly-acquired. But it changes the image_source parameter to "file". You will find the meaning of the image_source parameter and its values described here. Change image_source back to "daq" to acquire fresh data from the data recorder.

Recorder Script

If you press the Info button in the Recorder Panel, the Info Window opens with an array of Recorder parameters and their current values. These parameters are the elements of the LWDAQ_info_Recorder array. The Recorder uses these parameters to control data acquisition, analysis, and its own windows. Feel free to change them. The worst thing that can happen is the program freezes and you have to re-start it.

Press the Script button in the Info window and a text window will open up showing you the TCL script that defines the Recorder Instrument functions. All LWDAQ instruments define themselves with a few standard routines. In this case the standard routines are LWDAQ_daq_Recorder, and LWDAQ_analysis_Recorder. The former acquires a new image from the data acquisition system, the latter applies Recorder Instrument analysis to an image. The additional routin LWDAQ_create_controls_Recorder defines the Recorder Instrument's custom controls. These are the display entries and the reset button. The function of the reset button is defined by LWDAQ_reset_Recorder.

If you want to learn how, in detail, the Recorder Instrument communicates with the data recorder device, look at the script for LWDAQ_daq_Recorder. You will see lines that open a socket to the LWDAQ driver over TCPIP, write to the command word on the data recorder, and download blocks of data. The basic act of downloading data from the data recorder into the LWDAQ driver's RAM, and then from the driver RAM to the LWDAQ software, is straightforward. What complicates the Recorder acquisition is the buffering required to parse the data recorder message buffer contents into image periods, and to do so with minimal delay.

Message Display

Nowhere in the Recorder Instruments parameters will you see any mention of the data recorder clock speed. Nor will you see any mention of the relationship between the sixteen-bit message values and any physical quantities. The Recorder Instrument is a generic downloader and sorter of sixteen-bit values stored in a generic data recorder.

The message display uses a different color to display each channel. With analysis_channels set to its default value of * (which means "all messages"), the Recorder Instrument displays all messages in the current image (as we say in the previous section: in LWDAQ terminology, a block of messages assembled and returned by the Recoder Instrument is an image). The message display is the product of analysis of the image. Whether you display all the channels or not, they remain in the image. The Recorder Instrument will not delete them from the image.

Each channel number has its own color. The colors are unique, but it's unlikely that you could tell them apart if you had messages from all 256 possible channel numbers displayed at the same time. Subcutaneous transmitters us only 196 of the 256 available channels for their data transmissions.


Figure: Channel Color Numbers Used By The Recorder Instrument.

The horizontal axis of the display covers the image time. The Recorder Instrument does not know how to translate daq_num_clocks into image seconds, but it does know how to translate the clock messages and message time stamps into time in units of data recorder clock ticks. So it plots the message values against the time, in clock ticks, at which the data recorder stored the message. If there are missing messages, you will see a straight line joining the messages on either side of the missing messages.

There are three ways to plot the voltage in the Recorder Instrument window. The simple plot we select with the SP button. The range sets the range of the plot from bottom to top in ADC counts. The offset sets the voltage at the bottom of the display. The centered plot uses range in the same way, but ignores the offset. The plot of each signal is centered upon the window, so that the average value of the signal is exactly half-way up. The normalized plot ignores both range and offset and fits the signal exactly into the height of the display. The horizontal axis is time. The full range from left to right covers the recording interval. No matter what you do with the display, the data in the image is unaffected.

Channel Selection

If you have many active data channels, you may wish to isolate one or more of them for display by the Recorder Instrument. The first stage of channel selection takes place in the data receiver. When we Reset and Configure the data receiver, we configure it to receive from one of fourteen sets of channel numbers (sets 0-13) or from all sets. Set zero (0) contains channel numbrs 1-14, set one (1) numbers 16-30, and set thirteen (13) numbers 208-222, as we describe here. With set_num equal to the set we want to select, or with set_num equal to "*", we configure the data receiver to record from either for one set or all sets.

The second stage of selection takes place in the Recoder Instrument, by means of the analysis_channels string. If this is set to *, the Recorder Instrument analyzes and displays all channels in the image. If it is set to a list of channel numbers separated by spaces, only these channels are analyzed and displayed.

Example: There are twenty Subcutaneous Transmitters (A3028) operating in range of our Octal Data Receiver (A3027). Ten belong to set zero and ten belong to set six. We can record only from the ten in set zero with set_num = 0 when we reset and configure the data receiver. After that, we may wish to isolate the traces from channel numbers 3 and 5, which we do with channel_select = "3 5". Alternatively, we can configure the data receiver to record from all sets. Now we download the messages from all twenty transmitters.

When we configure a data receiver to record from only one set, we do not get data from any other set. When we configure it to record from all sets, all messages are stored to its message buffer. In both cases, the data receiver eliminates almost all duplicate messages that arise when the same messages is received on multiple antennas. The Recorder Instrument purges duplicates from its data as soon as it receives them from the data recorder. We can turn off the purge by setting purge_duplicates to zero. Because of the purge, the data collected by the Recorder Instrument, and passed on to tools such as the Neuroarchiver will contain no duplicate messages.

The result string returned by the Recorder Instrument changes with the value of analysis_channels. When you select all channels with *, every channel with one or more messages is listed in the line. When you select a specific subset of channels, only these channels, in order of ascending channel number, appear in the result string. At the end of this specific list of channels, the Recorder Instrument adds an entry for channel number −1, which is "all other non-zero channels", and gives you the number of messages in these other channels.

Message Analysis

By default, the Recorder Instrument analyzes every new data image it acquires. The analysis consists of plotting, listing, and summarizing the acquired data. You can turn off analysis of the image by setting analysis_enable to zero. With the analysis turned off, you will see the bytes of the messages drawn as individual pixels in the top part of the image. When you enable the analysis, you get the message value plots. The analysis does not change the image data.

The Recorder Instrument's image analysis routine is called LWDAQ_analysis_Recorder. This routine calls lwdaq_recorder, which in turn calls lwdaq_A3007_recorder. The lwdaq_sct_recorder routine is defined in a Pascal source filed called electronics.pas. If you want to learn about the analysis in detail, as a guide to writing your own reconstruction program, take a look at the Pascal file, where the algorithm is explained in detail in the comments.

The lwdaq_recorder command is available at the LWDAQ console. Its arguments are an image name and a command string. The command string begins with an instruction. The following table lists the lwdaq_recorder instructions.

InstructionFunctionUsed By Recorder
listLists active channels in an image.Yes
printPrints message contents and lists errors.Yes
plotPlots messages in the image overlay and returns summary result.Yes
clocksCounts clock messages and locates specific clock messages.Yes
extractExtracts messages from a single channel.No
reconstructElminates bad messages and adds substitute messages.No
purgeRemoves duplicate messages.Yes
Table: Recorder Analysis Instructions.

The purge instruction is the only instruction that alters the data downloaded from the data receiver. It eliminates duplicate messages, whether they are consecutive in the data or not. All other instructions leave the data in the unaffected.

The Recorder uses the plot instruction to plot the channels in the image overlay and to generate the summary string. It uses the list instruction to obtain a list of active channels. Internally, as it is accumulating an image with the correct number of clock messages, the Recorder uses the clocks instruction to determine how many clock messages are in a new packet of data, and where the first and last clock messages are in the data. When the Recorder encounters errors in acquired data, it uses the plot instruction to obtain a list of these errors and their locations within the data, so that it can attempt to remove the errors and correct mis-alignment of message boundaries in the data recorder. Such errors occur during rare electrical events such as static discharge on the data acquisition hardware or power interruptions. You can see for yourself how the Recorder uses the lwdaq_recorder instructions in the Recorder Instrument source code.

You can apply the lwdaq_recorder instructions yourself, using the console and the name of the current Recorder image. After you acquire a Recorder image, look at the value of memory_name. Suppose it's Recorder_1. Go to the File Menu and open the console. Enter the following at the console prompt.

(LWDAQ) 1 % lwdaq_recorder Recorder_1 "clocks"
0 128 3166 

The result we show is the one we get when we apply clocks to this image. There are 0 errors in the clock sequence in this image, there are 128 clock messages, and 3166 messages in total. Each message in an image has a message index, which is its position in the sequence of all messages. The first message has message index zero (0). Each message also has a channel index, which is its position in the sequence of messages in its channel. The following command returns the message index of each of three clock messages. The command specifies the clock messages by channel index.

(LWDAQ) 2 % lwdaq_recorder Recorder_1 "clocks 0 64 128"
0 128 3166 0 1583 -1  

Clock message zero also has message index zero. The first message in the image is a clock message. Clock message 64 has message index 3166. There is no clock message 128, so the command returns −1 to indicate the absence of the specified clock message.

The print instruction instructs lwdaq_recorder to print out the messages for us, as shown below. Each message gets a separate line. On each line is the message index (the 0'th message is the first one in the image), the channel number, the data value, and the timestamp. After that comes the hexadecimal representation of the four bytes that make up the message in the block of data we acquired from the data recorder.

(LWDAQ) 3 % lwdaq_recorder Recorder_1 "print 0 10"
Data Recorder Firmware Version 5.
Total of 3166 messages and 128 clocks.
Encountered 0 errors in data block.
Messages 0 to 10 (index id value timestamp $hex):
    0   0  1281   5 $00050105
    1   5 42860   8 $05A76C08
    2   6 40972  18 $06A00C12
    3   9 40654  22 $099ECE16
    4   3 30275  33 $03764321
    5   7 37119  37 $0790FF25
    6   4 46759  60 $04B6A73C
    7   5 43183  72 $05A8AF48
    8   6 41065  73 $06A06949
    9   9 40063  87 $099C7F57
   10   3 30456  97 $0376F861
End of Messages

Here we see print used to print out the first eleven messages. If we don't specify the message range, print will print the contents of all the messages. The first message is a clock message. It's channel number is zero and its timestamp is 4. The timestamp of a clock message gives us the data recorder firmward version, which is in this case 4. In the following print command, we see the messages and payload bytes produced by an animal location tracker, A3032A. The A3032A produes sixteen payload bytes. The first fifteen are power measurements from its fifteen detector coils. The last byte is the A3032A firmware version number.

Data Recorder Firmware Version 39.
Total 4014 messages, 128 clocks, 0 errors, and 0 null messages.
Messages 0 to 10 (index id value timestamp $hex):
    0   0 45363  39 $00B13327 1B221D2D1C2324161E1F181B1E171501
    1   4 45782   8 $04B2D608 645E4B4D613A595D4B213D505C305401
    2   1 44397  22 $01AD6D16 68553B6B6E3954534225363753195301
    3   5 43957  25 $05ABB519 67645F82565D5E494E3F2B4D3C363C01
    4   7 43985  31 $07ABD11F 60734E6D8B1A496B3742354D4D384501
    5   3 45940  40 $03B37428 635E4A4C6139585C4B213E4F5B2F5301
    6   8 44793  45 $08AEF92D 68646283785C6470464D444E48414C01
    7   2 45021  48 $02AFDD30 61776A7C865769422A54293B504A5D01
    8  13 43951  50 $0DABAF32 6A6C697887525F31454B284A5E4A5B01
    9   4 45759  71 $04B2BF47 645E4B476338585E4C203A505D334C01
   10   1 44387  89 $01AD6359 6955416E6C3E515643213C3252235601
End of Messages

If you want to extract messages from a single channel, and get them in a format that's easy for plotting on the screen, or pasting into a spreadsheet, use the extract instruction. Here is an example, obtained from our Recorder_hum.daq data in image Recorder_1. We use the ellipsis ("...") to indicate lines we have cut from the console output.

(LWDAQ) 4 % lwdaq_recorder Recorder_1 "extract 1"
18 40972
73 41065
135 40923
208 40652
265 40586
...
32525 44508
32593 44287
32658 44226
32708 44208

The extract instruction takes one parameter: the channel number you want to extract from the image data. The extract result does not contain the channel number. It returns the time and value of each message on a separate line. The time is in clock ticks since the start of the image interval. You can cut and paste the extract result into a spreadsheet and plot the values that way, or you can use LWDAQ library routines to plot the same data in a LWDAQ window.

Each LWDAQ image provides an overlay in which you can draw graphs. The following lines use extract to extract channel five (5), plot it in the overlay using lwdaq_graph, and refresh the display of Recorder_1 in the Recorder Instrument window using lwdaq_draw.

(LWDAQ) 5 % lwdaq_graph [lwdaq_recorder Recorder_1 "extract 5"] Recorder_1 -fill 1
(LWDAQ) 6 % lwdaq_draw Recorder_1 recorder_photo

The plot instruction plots messages in an image's overlay in the same as lwdaq_graph. Here we tell lwdaq_recorder to plot messages from channel three (3) and five (5) in the overlay.

(LWDAQ) 7 % lwdaq_recorder Recorder_1 "plot 0 65535 DC 3 5"
3 511 42461.0 3256.6 5 482 43415.2 522.4 -1 2045
(LWDAQ) 8 % lwdaq_draw Recorder_1 recorder_photo

The "0 65535" means that the display should show data values in the range 0 to 65535. The "DC" means that the coupling for the display should be direct. With "AC", the analysis will subtract the average of each channel's data value before it plots. The "3 5" means plot channels 3 and 5. If you say "*" instead of giving channel numbers, lwdaq_recorder will plot all available channels. In the results string we see that channel 3 provided 511 samples with average value 42461.0 and standard deviation 3256.6. Channel −1 is for "all other channels except channel zero". Channel zero (0) is, of course, the clock message channel. There were 2045 samples in the other channels.

The reconstruct instruction attempts to eliminate bad messages from the message stream and replace missing messages with substitute messages. The value of a substitute message will be the value of the previous message in the message stream.

The reconstruct instruction's task is complicated by the fact that most sources of messages, such as the Subcutaneous Transmitter (A3028A) displace the instant of each individual message transmission by a random time. We call this random displacement transmission scatter. Transmission scatter makes it possible for two message sources with the same period to share the same transmission medium without corrupting a significant fraction of one another's messages. We discuss transmisison scatter in more detail in Collisions.

Example: Two Subcutaneous Transmitters (A3013A) share the same 900-MHz ISM radio-frequency band. Each A3013A transmission takes 7 μs. A 32.768 kHz clock runs in the transmitter. Every 64 clock ticks, the A3013A prepares to transmit a message. But it delays transmission by 0 to 15 clock ticks at random, which is what we call transmission scatter. The A3013A uses the lower four bits coming from its digital to analog converter as the random number. These bits being dominated by random noise, they make a good random number generator. When two A3013As drift into synchronicity, so that the 64-tick sampling periods begin within a few microseconds of one another, transmission scatter guarantees that only one in sixteen messages will be collide.

With transmission scatter, each message transmission occurs at some random time within a transmission window centered upon a nominal transmission time. The nominal transmission times, and therefore the transmission windows, are spaced by an exact and well-known transmission period.

A reconstruct command requires that we specify one message streem for reconstruction by giving its channel number. We specify the period and scatter of the message stream as well. The only units of time used by lwdaq_recorder are data-recorder clock ticks. So we must specify the period and scatter in units of clock ticks. We specify the scatter as the maximum displacement of the transmission time from the nominal tranmission time.

Example: A Subcutaneous Transmitter (A3013A) transmissions occur somewhere within 480 μs of the rising edge of its 512-Hz message clock. The distribution of the transmission time is random and uniform. The nominal transmission time is the center of this distribution, and is therefore 240 μs after the rising edge of the message clock. The transmission scatter is ±240 μs on either side of the nominal transmission time. The Data Receiver (A3018A) that receives our A3013A messages has a reference clock of 32.768 kHz. Each clock tick is 30 μs. The A3013A transmission scatter is therefore 240 / 30 = 8 clock ticks. The A3013A's transmission frequency is 512 SPS, so its period is 64 clock ticks. The time between two consecutive messages could be as little as 48 ticks and as great as 80 ticks, or 1.5 ms to 2.4 ms. You can see transmission scatter between consecutive A3013A messages here.

Here is an example reconstruct command, applied to Recorder_hum.daq example image in Recorder_1.

(LWDAQ) 1 % lwdaq_recorder Recorder_1 "reconstruct 1 64 8 30000"
-7 30000
59 41175
121 44466
183 47057
246 46943
317 44541
380 40901
432 38665
498 39453
...
32693 38335
(LWDAQ) 2 % 

We see the reconstruction of channel 1 with period 64 ticks and transmission scatter 8 ticks. The fourth number in the command is the standing value, which we set to 30000. The standing value of the channel is the value we want reconstruct to use if it determines that there is a missing message at the beginning of the image's message stream. In this case, reconstruct finds that there is a transmission window overlapping the beginning of the image's time interval. The center of this transmission window is −7 clock ticks before the beginning of the images's time interval, and therefore overlaps this time interval because the transmission scatter is 8 clock ticks. The reconstructed message stream contains a message at time −7 with value equal to the standing value we specified.

A reconstruct command allows us to apply reconstruction to a continuous message stream. Each time we call reconstruct we apply it to one image only, but we pass information to it from previous images using the command string. The standing value is the value of the final message from the previous image. The final message value in our example reconstruct result is 38335.

Suppose reconstruct gets to the end of a message stream in an image and finds that the final transmission window overlaps the end of the image time. This happens in our example above. The final transmission window is centered at time 32761 clock ticks. Our extract command returned a final message "32765 38205", which lies in this window. You will note that our reconstruct command did not return this final message. It did not return the message because it cannot be confident of the status of the message. What if it is a bad message, and the valid message for that transmission window occurs at the beginning of the next image time interval? In the general case, there can be one or more messages within the final transmission window, and reconstruct must somehow pass these messages to the next reconstruct command that will be applied to the next image in the message stream.

The extract and reconstruct commands both use the image results string to store meta-data, such as a list of unaccepted sample values. The image results string is part of the image data block, as described here. The extract command sets the image results string to the number of clocks in the image followed by the number of messages the command extracted from the selected channel. The reconstruct command writes four or more numbers to the image results string.

(LWDAQ) 2 % lwdaq_image_results Recorder_1
128 512 4 5 38205 

The lwdaq_image_results command returns an image's results string. We obtain the above result if we call lwdaq_image_results immediately after our call to reconstruct. The first number is the number of clocks in the image. Then comes the number of messages in the reconstructed message stream. The third number is the number of messages reconstruct rejected as bad messages. In this case, it rejected 4. The fourth number is the number of missing messages for which reconstruct generated substitutes. In this case, it inserted five substitute messages. The final number is the value of the unaccepted message we discussed above. If there were several such unaccepted messages, the image results string would contain their values one after another. In this case, we have only one value, 38205.

We pass these unaccepted values to the next reconstruct simply by appending them to the end of the command string, like so:

(LWDAQ) 1 % lwdaq_recorder Recorder_2 "reconstruct 1 64 8 38335 38205"

Here we assume that the name of the next image, one it has been acquired, is Recorder_2. We specify 38335 for the standing value, because that was the value of the last message returned by reconstruct when it operated upon Recorder_1. At the end we give the single unaccepted value.

In our first reconstruct command, we did not pass the correct standing value, nor any unaccepted values. That's why we ended up with the "−7 30000" sample, which cannot be real. As you can see, reconstruct does reconstruct a message stream, but there may be one more or one less messages in each section of the message stream it returns.

The extract and reconstruct routines create a list of samples and their payloads for subsequent use by routines such as lwdaq_locator command. The locator command takes the payload and extracts from it coil power measurements which it uses to measure the location of a subcutaneous transmitter on a tracker platform.

Error Correction

It is possible to receive corrupted data from a data recorder. The most common cause of such corruption is static discharge or electrical power glitches. The corruption usually takes the form of extra bytes inserted into the data, but may involve missing bytes also. Another form of corruption occurs when the Recorder Instrument cannot keep up with the data produced by the data recorder, such as when the network connection is slow. The message buffer on the data recorder overflows and the Recorder Instrument sees a block of data go missin.

Starting with LWDAQ 7.5, the Recorder Instrument source code contains a routine LWDAQ_error_fix_Recorder that finds and removes additional bytes in the acquired data. There is nothing we can do about missing clock messages. The Recorder Instrument performs as much error correction as possible before it resorts to resetting the data recorder. When we reset the data recorder, we reset the clock value, which confuses the Neuroarchiver's calculation of time within an archive.

In the Neuroarchiver we can use the Recorder Instrument's analysis routines to display errors in the messages recorded to disk, as they are after error-correction.

Data Processing

The Neuroarchiver Tool uses the Recorder Instrument to obtain data from a data recorder. The Neuroarchiver uses the lwdaq_recorder routine's the reconstruct instruction to clean up and patch up the message stream received from one or more sources. It stores raw data, reconstructed data, and fourier-transformed data to disk. The Neuroarchiver is intended for use with Subcutaneous Transmitters and Data Receivers, as described here. Other data recorder systems will require their own tools. If you plan to write such a tool, start by looking over the Neuroarchiver.tcl.