Telemetry Control Box (TCB)

Contents

Description

[19-DEC-22] The Telemetry Control Box (TCB-A) is a telemetry receiver and location tracker for telemetry devices such as our Subcutaneous Transmitters (SCT), Implantable Inertial Sensors (IIS), or Implantable Stimulator-Transponders (IST). The TCB-A16 provides sixteen antenna inputs, to which we can connect up to sixteen coaxial antennas, such as the Loop Antenna (A3015C). Each antenna input is equipped with its own dedicated receiver and power meter. The telemetry messages we record with a TCB contain not only the telemetry channel number and sample value, but also the identity of the top antenna, which is the antenna at which the message presented the greatest microwave power, and the top power, which is a logarithmic measure of the microwave power at the top antenna. The antenna that receives the most power from a transmitter is most likely to be the antenna to which the transmitter is closest. so the top antenna and top power allow us to estimate the location of all transmitters in a telemetry system. All versions of the TCB connect directly to Power over Ethernet (PoE), from which it obtains power and through which it performs all communication.

Figure: Telemetry Control Box, Front View. On the back side are sixteen BNC sockets for antenna connections and a single RJ-45 socket for Power over Ethernet (PoE).

The TCB-A16 replaces the Octal Data Receiver (A3027E, ODR). It is built using circuits developed for the Animal Location Tracker (A3038, ALT). The TCB-A16 provides sixteen antenna inputs, the ODR provided eight. The TCB-A16 provides antenna power measurements, the ODR did not. Unlike the ODR, the TCB-A16 does not require a LWDAQ Driver (A2071E). If we combine the TCB-A16 with the synchronous video provided by our Animal Cage Cameras (AACs), the TCB-A16's location tracking allows us to determine which animal is which in the video.

Figure: Display Panel. Activity lamps glow in proportion to sample rate. In this picture, the Detector Module Error (DMERR) light is shining, which indicates a problem with telemetry reception. This light should never turn on during normal operation of the TCB. Press SHOW to turn on all the lights temporarily. Press HIDE to turn off the activity lights if they are too bright.

Our planned Telemetry Control Box Version B (TCB-B) will provide a command transmitter for every antenna connection, so that we can use any of its antennas to transmit commands devices capable of receiving commands, such as our Implantable Stimulator-Transponders (ISTs). For reliable reception by an IST, we can first arrange for the IST to transmit a message of its own, then choose the top antenna for that message to transmit our command. The TCB-B16 provides sixteen antenna inputs. We select the best antenna in the following way: we transmit an XON command through all sixteen antennas consecutively so as to make sure our target IST is transmitting its synchronizing signal. The synchronizing signal is just like an SCT signal, so the TCB-B16 will tell us which antenna is receiving the signal with the greatest power, and we choose this antenna to transmit a subsequent stimulus command. The TCB-B16 replaces the Command Transmitter (A3029) and LWDAQ Driver (A2071E), combining command transmission and telemetry reception into one instrument with one PoE connection, greatly simplifying our data acquisition and control system.

Versions

The following versions of the Telemetry Control Box (A3042, TCB) exist.

The following sub-assembly versions exist.

Design

[28-SEP-22] We make the TCB-A16 by taking an Animal Location Tracker (ALT) Base Board (A3038BB-D2), putting it in a box, populating it with ALT Detector Modules (A3038DM-B2), connecting the detector modules to BNC sockets a Sixteen-Way Filtering Feedthrough (A3042FF-A16) on the back wall of the box, connecting a Display Panel (A3042DP-A) to the base board, and fastening the display panel to the front of the box. We connect the ALT's RJ-45 socket to an RJ-45 feedthrough on the back wall of the box. The TCB comes with an Ethernet jumper cable so we can connect it to a PoE switch. We order up to sixteen Loop Antennas (A3015C) separately, depending upon how many we need to deploy in our telemetry system. The TCB-B16 is identical to the TCB-A16 except its antenna connections pass through a Sixteen-Way Transmitting Feedthrough (A3042TF-A16) on the back wall. The TCB now has available sixteen independent command transmitters as well as four independent digital input-outputs accessible on the back wall through BNC connectors.

The P3042BB firmware is for the logic chip on the base board. This firmware is similar to the ALT Base Board firmware except for the addition of communication with the display board and transmitting feedthrough. We use C3038BB for the RCM6700 module, same code as for the ALT LWDAQ Relay. We use P3038DM on the detector modules, same code as for the ALT Detector Modules. We use P3042DP on the display panel and P3042TF on the transmitting feedthrough.

If you want to control an A3042 with your own data acquisition software, consult the LWDAQ Spectification for details of the TCPIP messages we use to communicate with the A3042. The A3042 acts like a LWDAQ Driver for the purpose of data acquisition. Its controller address space is defined in VHDL as follows.

constant cont_id_addr : integer := 0; -- Hardware Identifier (Read)
constant cont_sr_addr : integer := 1; -- Status Register (Read)
constant cont_djr_addr : integer := 3; -- Device Job Register (Read/Write)
constant cont_hv_addr : integer := 18; -- Hardware Version (Read)
constant cont_fv_addr : integer := 19; -- Firmware Version (Read)
constant cont_crhi_addr : integer := 32; -- Command Register HI (Write)
constant cont_crlo_addr : integer := 33; -- Command Register LO (Write)
constant cont_cfsw_addr : integer := 40; -- Configuration Switch (Read)
constant cont_srst_addr : integer := 41; -- Software Reset (Write)
constant cont_fifo_av_addr : integer := 61; -- Fifo Blocks Available (Read)
constant cont_fifo_rd_addr : integer := 63; -- Fifo Read Portal (Read)

We memory portal address is 63 (0x3F), as in all LWDAQ controllers. But the A3042's memory portal is read-only. The A3042 does not support the stream_write instruction. It does, however, provide a stream_read, and it is the stream read that we use to download telemetry data from the A3042's controller memory. The telemetry data is stored in a first-in, first-out (FIFO) buffer in the controller. To operate an ALT, the data acquisition software begins by writing any value to the Software Reset location (41), which resets the message clock, clears the message buffer, flashes the white lights, and configures the detector modules. After that, keep reading the Blocks Available location (61). Multiply the blocks available by 512 to get the number of bytes available. The maximum value of this counter is 255, so if there are more than 130 kBytes of data available, you will not know it. The ALT data is divided into six-byte message. Each message begins with a four-byte SCT record, which we describe elsewhere. After that comes the top power and the top antenna, each one byte. When we download from the ALT, we download a whole number of messages, so the number of bytes should be divisible by twenty. We now execute a stream_read and download the number of bytes we expect.

If you want to configure the TCB to record only from certain SCT channel numbers, you can do this by writing commmands into the controller using the Command Register (two bytes) and the Device Job Register. You will find Tcl code for configuring ALTs and TCBs in Receiver.tcl.

Set-Up

[27-JUL-22] The TCB obtains all its power and communication through a single Power-over-Ethernet (PoE) socket. The TCB system consists of the TCB, a PoE switch, coaxial cables, feedthroughs, and Faraday enclosures. In the paragraphs below, we provide detailed instruction on setting up the TCB for communication with your computer, but ultimately refer you to the SCT setup instructions once that communication is established.

Power Up PoE Switch: Connect power to your PoE switch. If you are not in the United States, you will need a computer power cable to connect to your local type of wall power socket, but you can be assured that the power adaptor will operate with any AC voltage 100-250 V, 50-60 Hz. Lights should illuminate on the PoE switch.

Connect TCB to PoE: Use a network cable to connect the TCB to the PoE switch. This cable can shielded or unshielded, and it can be up to ten meters long. Lights should illuminate on the TCB.

Install Softrware: Download and install the latest version of the LWDAQ software from here, following these installation instructions.

Connect Computer to PoE: Connect your data acquisition computer to the switch. Configure your computer to use its wired Ethernet interface with subnet 10.0.0.0 and IP address 10.0.0.20. The TCB ships with an IP address 10.0.0.x, where x is given by the last two or three digits of the serial number on the back of the TCB enclosure. The table below gives examples of serial numbers and their addresses.

You should see a link light beside the sockets you have used on the PoE switch for your computer and your TCB. Launch the LWDAQ software and try to contact the TCB using the Configurator Tool, as described here. Don't proceed until you can contact the TCB and read its EEPROM.

Continue with Telemetry Setup: Connect antennas, set up software, and test out sample transmitters with the help of our Subcutaneous Transmitter (SCT) setup instructions, which you will find here.

In the Neuroplayer, if you can use the Neurotracker to see which antenna is receiving the most power from each of your transmitters.

Power Consumption

[27-JUL-22] The TCB-A16 and TCB-B16 consume a maximum of 8 W. The TCB-B16 peak power consumption during command transmission is 10 W.

Modifications

[18-NOV-22] The A3038BB-D2 needs the Detector Module Clock Modification, see A3038 Modifications, where we add C48 = 100 pF to the base of DMCK to reduce reflections off the end of the daisy chain.

[24-DEC-22] The A304201A PCB for the display panel needs the following. Test points must be marked on back side of board. Need test points for detector control bus signals, marked on both sides. Two more mounting holes to stop the board from flexing.

Development

[07-MAR-22] Nathan reports as follows. "The system is a simple setup of two plastic cages taped together with an antenna on opposite sides. The antennas are spaced 70cm apart. I turn off all the detector modules except for the two that are taped to the sides of the cages. I turn on a transmitter inside the faraday canopy and move the transmitter around the center of each cage, changing position and orientation. I notice that when the transmitter is moving in the center of the cage (about 3 times closer to one antenna than the other) the tracker in the Neuroplayer displays the transmitter being on the correct antenna approximately 95% of the time. That is to say that in this two-antenna system, if an animal was moving in the center of a cage there is a 95% chance that our system would detect that mouse being in the correct cage."

Figure: Animal Location Tracking with Loop Antennas. We have two A3015C antennas on either side of two animal cages. Antennas are connect to detector modules on an A3038C ALT.

[21-MAR-22] We are comparing the Telemetry Control Box to the Octal Data Receiver. Nathan writes, "I'm still working on what went wrong with the coaxial combiner, but I was able to get some reception readings that are consistent with past ones so I went ahead and compared the two receptions. I began by only connecting one antenna and comparing reception on different types of transmitters. My results are as follows: For the Faraday canopy test transmitter that is taped onto the end of a stick, I got 75% reception using the ODR as well as the TCB while moving the transmitter to various places in the canopy. Using the rat transmitter, I got 85% reception on both receivers. Similarly, I got 85% reception on both receivers using the pup transmitter with a worse antenna and 90% reception on both receivers using the pup transmitter with the better antenna. None of the transmitters gave different amounts of reception on one receiver than the other. To confirm this, I did another experiment where I moved around each of the transmitters individually while the coaxial combiner output was sent to a T-junction that split to both receivers. This way I could see how much reception I am getting from both receivers at the same time. I never encountered a situation in which I was getting reception on one receiver and not on the other."

Here we see that, by splitting the antenna signal and sharing between the two receivers, we can watch the reception lights provided by both receivers to see if they are synchronous. We find they are: reception by the ODR implies and is implied by reception by the TCB, even though the ODR and TCB do not use the same detection method. The ODR uses a hetrodyning receiver with active demodulator. The TCB uses a direct, narrow-band amplifier with split-capacitor tuner and crystal diode demodulator.

[27-JUL-22] The Display Panel (A3042DB-A) printed circuit board A304201A is almost ready for fabrication. We begin work on the Filtering Feedthrough (A3042FF-A16) printed circuit board A304202A. We will be using an ALT Base Board (A3038BB-D), but we will remove its antennas and we will program the board with firmware P3042BB that re-assigns detector control bus lines DC5 and DC6 for use by the display panel through the flex connector at the end of the detector module daisy chain. We will likewise re-program the A3038DM-C detector modules with firmware A3042DM so that they turn on their lights with DMRST and have no HIDE function, which we don't need inside the TCB box.

The base board will use DC5 to transmit to the display board. The display board will use DC6 to transmit to the base board. We still have TP1, TP2, TP3, and TP4 free on the base board to use for communication with the trasmitting feedthrough. The base board logic will use DC5 to transmit eight-bit serial messages. The first four bits will be an operation code 0-15. The next four bits will be an operand 0-15. Here are the messages the base board will be able to send to the display board.

Here are the messages the display board will send to the base board.

The transmitting feedthrough will need to receiver serial messages also, so as to select one of sixteen antennas and prepare them for power transmission. The switching of the power itself for command transmission to crystal radios we wil perform with a dedicated logic line that enables the chosen RF oscillator when it is asserted. We expect to use TP3 for serial selection of antenna and TP4 to turn on RF power. The transmitting feedthrough's logic will disable command transmission automatically after a short silence, perhaps as short as 10 ms.

[22-AUG-22] We have the first A3042FF-A16 assembled and tested. We have the A304201A and are starting assembly. We note inconsistent masking of vias.

Figure: Inconsistent Via Masking on A304201A.

[07-SEP-22] The detector control bus on the base board consists of eight logic lines. Their functions in the A3042BB will be as follows.

The display panel will receive serial messages on DC6 (SDO) and transmit serial messages on DC5 (SDI). The protocol assumes that both the base board, detector modules, and display panel have pull-down resistors on SDO and SDI respectively. Timing of the protocol as shown below.

Figure: Serial Protocol for Communication Between Base Board and Display Panel.

When the Display Panel sees DC0 (DMRST) go HI, it will illuminate its RESET lamp and reset its state machines. When the Display Panel sees its RESET button pressed, it will assert DMRST, which will in turn reset its own state machines. The Base Board will detect the HI on DMRST when provided by the Display Panel and assert its own hardware RESET. When the Display panel sees its CONFIG button pressed, it will send a CONFIG message to the Base board. When the Display Panel sees DC2 (DMERR) HI, it will illuminate its DMERR lamp.

[15-SEP-22] Create P3042BB repository for base board firmware and software. We start with version 1.0 based upon P3038BB v8.3. We change firmware version to 1, change receiver type to 120 so the Neurorecorder can identify the TCB. Now have v1.1. Begin adding code for communication with display panel. Plan is to leave the detector modules as they are, and use SHOW and HIDE lines for communication.

[16-SEP-22] We have P3042BB v1.2 transmitting display messages on SDO = DC6. When the base board's SHOW button is not depressed, serial communication is as shown below, with LO before and after.

Figure: SDO During Transmission to Display Panel. Blue: DC6 at start of detector bus. Green: DC6 at end of detector bus. The SCT is channel No228, which has remainder 4 when divided by sixteen.

[19-SEP-22] Consider consolidating the P3042BB and P3038BB code into one program that detects the hardware version through TP1 to HI for A3043 and LO for A3038. We anticipate further development on the P3042BB code as we add the transmitting feedthrough and auxiliary logic inputs and outputs. These developments would have to be tested not only in the A3042 but also the A3038, even when the enhancements have no affect upon the A3038 behavior. We will continue with separate repositories, despite the similarity between the firmware. Consider adding feature to P3042BB whereby we can discard the tracker payload before download. Maximum data rate from a TCB with 40 × 512 SPS transmitters is 400 kByte/s. If we can discard tracking, data rate drops to 80 kByte/s. With a discard tracking option, we could operate the TCB like an ODR. This feature proves difficult to incorporate into our existing software and difficult to debug. We abandon the idea. The maximum data rate of the RCM6700 is 1400 MBbyte/s so we don't expect to have trouble getting the data to our computer. We could strip the payload from the data before storing to disk, or remove from disk files some time after recording to compress. Discarding the data at the receiver means irrevocable loss of information. Tag firmware v1.3.

[28-SEP-22] We order the A304203A PCB, a new version of the feedthrough PCB, this one with no components other than the connectors, signals passed straight through with transmission lines, including the Ethernet. This will be the PCB we use in the Sixteen-Way Straight Feedthrough (A3042SF-A16). We decided there is no point in using the filtering feedthrough with our new detector modules, these new modules being equipped with the SF2908E filter present on the filtering feedthrough.

[16-NOV-22] We have six TCB enclosures made to our drawings. We are assembling all six as TCB-A16.

[20-NOV-22] We re-design the detector module readout so that the detector modules operate independently. We create a new repository for the TCB detector module firmware P3042DM. Start with v1.1 identical to A3038DM v8.1, then eliminate INCOMING and RECEIVE, replace with Message Ready (MRDY) on DC3 which indicates that one or more detector modules have a message withing in their message buffer, including ID, sample, and power. The base board firmware we modify to read out the detector modules in five steps, the first four retrieving the ID, sample, power, and daisy chain index of the message in the first detector module asserting MRDY in the daisy chain. We obtain this index by driving a zero onto the daisy chain data bus at the readout module, and incrementing this byte by one each time it passes through a downstream module. Instead of reading out the daisy chain in an interrupt, we read it out in the main loop. By this means, if we unplug a detector module, leaving MRDY asserted continuously by a detecor module that is not being read out, we are still able to maintain the indicator lights, while reading and rejecting the zero-identifier message we get from the absent module. The working code is A3042BB version 2.1. Right now, if multiple detector modules receive the same message, they all are read and written into the buffer with no payload: the power and antenna number are discarded. Assuming the none of the messages is corrupted, they will all be identical.

[21-NOV-22] We with to store the power and antenna number as a payload of two bytes attached to each clock and sample message. The clock message payload can be two zeros, but could in the future hold background power. Suppose we read out all messages, save them to the message buffer, and upload them over TCPIP with our RCM6700 Relay. In our lwdaq_receiver routine, we can purge all but the most powerful copies of the same sample, leaving the one with the highest power, along with its antenna number. Thus we can determine the approximate location of the transmitter, and choose the best antenna for command transmision to implantable stimulators. But our maximum data rate with 40 × 512 SPS transmitters, if they should all be received simultaneously by all sixteen detector modules, is 2 MByte/s, which is greater than the 1.4 MByte/s maximum transfer rate of our RCM6700.

We develop the Reset Arbitrator so that it can cooperate with the Reset switch on the Display Panel. We want the Display Panel switch to cause a global hardware reset through DMRST. We define three levels of reset: Top-Level, Mid-Level, and Low-Level. The Top-Level arises when we press the red reset button on the base board, or the red reset button on the display panel, or we first connect power. The Top-Level Reset resets the RCM6700 TCPIP interface (the Relay), the base board logic (the Controller), the detector modules, and the display panel. A Mid-Level reset is initiated by the Relay or the Controller. It resets the Controller, Detector Modules, and Display Panel. A Low-Level reset is initiated by the Controller. It resets the Detector Modules and the Display Panel.

[22-NOV-22] With 40 × 512 SPS, each sample transmission takes 10 μs, so we are transmitting for 20% of the time. The chances of any one sample colliding with any other sample is roughly 20%, so 80% of the time, we will have multiple copies of the same sample being read out of the detector modules with no other sample interrupting. When we see MRDY asserted, we read out a message A. If MRDY is no longer asserted after this readout, we store A in our message buffer. If MRDY is still asserted, we read out another message, B. We compare A and B. If their channel identifiers are the same, and the power of B is greater than the power of A, we over-write A with B. Otherwise we discard B and check MRDY again. But this means we will eliminate at least 80% of duplication, dropping the maximum required upload rate from 2 MBytes/s to 400 kBytes/s, which is well within the 1.4 MByte/s limit of the RCM6700 Relay.

We put the above idea to the test with A3042BB v2.2. We connect external loop antennas to two detector modules, place the antennas on top of one another, and set six transmitters on both of them, with a total of 4352 SPS. We have an average of 3888 messages per second, including the 128 SPS clock signal. We are not purging duplicates in the Receiver Instrument. We are seeing 15% loss in each channel. We separate the two antennas, leaving three transmitters on each. We see a similar number of messages per second. We are not seeing duplicates, but we are experiencing excessive loss. We find that MRDY always experiences a glitch 1.6 μs after its rising edge. We this from a solitary DM in the first position, or the last DM in a string of five. When the base board responds with DMRC before this glitch, we read out an invalid message from the DM and terminate the read. But MRDY remains HI, and we read again, this time we get a valid message.

Figure: Faulty Read from Detector Module, Pre-Glitch in MRDY. First DM in daisy chain, no others loaded. Green: MRDY 1 V/div. Yellow: DMRC 1 V/div. Blue: D2DS. Timebase 1 μs/div.

Any time the controller tries to read before the glitch, the read is terminated, as shown above. Any time the read occurs after the glitch, we obtain a valid message, as shown below.

Figure: Successful Read from Detector Module, Post-Glitch in MRDY. First DM in daisy chain, no others loaded. Green: MRDY 1 V/div. Yellow: DMRC 1 V/div. Blue: D2DS. Timebase 1 μs/div.

When we have five detector modules loaded, we observe the delay from DSD to DSU through each DM is 25 ns. We are forwarding the synchronized version of DSD to DSU, not the unsynchronized signal at the input pin. We fix this in the code. We note that the DS pulses are around 1 μs. We examine our code and count 20 clock cycles of execution time from rising to falling edge. The OSR8 is running at 20 MHz, 20 ÷ 20 MHz = 1 μs.

[23-NOV-22] Our MRDY glitch turns out to be the assertion of MRDY by another process within the base board firmware. We remove the errant code and MRDY is now asserted only when FIFO_EMPTY is false. Tag firmware P3042DM v2.2. Set up the main loop to transmit the the bottom four bits of the communication status register to the display panel every 256 cycles through the main loop, and to look for reception of the display panel configuration switch state with the same frequency, but offset. Have not tested with display panel, but sharing the CONFIGURE flag between the base board and display panel switches appears to be working. We re-configure and configure with the Configurator Tool. Reception from five detector modules is perfect. Tag firmware P3042BB v2.3.

The Controller deserializes SDI with a 2-MHz clock we call SCK in P3042BB. The SDI serial bit rate is 1 Mbps, as defined above. We synchronized SDI with the rising edge of SCK to make RSDI and with the falling edge to make FSDI, as shown below.

Figure: Double-Frequency Deserializer for SDI in Controller Logic. Yellow: when SDI changes just after a falling edge of SCK. Orange: when SDI changes just after a rising edge of SCK.

We use RSDI to detect the start bit. The state machine advances from state zero to one when it sees RSDI is HI. At the end of state two, we assign the state of FSDI to B7. The drawing above shows how the deserializer proceeds in two cases: when SDI transitions just after a falling edge, and just after a rising edge. In both cases, we see correct deserialization. We revive and adapt an existing Toolmaker script, Deserializer.tcl, which allows us to test the robustness of the above deserialization scheme in the presence of clock jitter and changes in CK frequency. According to the simulation, our deserializer is reliable with clock jitter up to 10 ns in our 500-ns SCK period.

Figure: Deserializer Simulation. Green: SDI. Blue: SCK. Red: RSDI. Yellow: FSDI.

[29-NOV-22] Firmware P3042BB v3.1 adds a two-byte payload to all messages. These are supposed to be the top power and top antenna. A bug in the v3.1 assembler code causes us to mistake which is the top antenna, but we fix this bug in v3.2 and add a prototype mapping between demodulator index and antenna socket that we can use to match the detector modules to their antenna feedthroughs. We add support for the TCB to the Receiver Instrument, Neurorecorder 158, and Neuroplayer 161. In the Neurotracker, we plot the sixteen antennas in a 4 × 4 array by default, and animal position is the location of the top antenna. We add lwdaq_tcb to our LWDAQ library to extract the top antenna and power from TCB data efficiently.

[06-DEC-22] Firmware P3042BB v3.1 contained a bug: the memory location used to store the new message and the previous message were the same. We correct this bug and find another bug in our message handling logic: previous messages with ID zero are being stored when they should not be. We make the following diagram of the logic to help us correct the code.

Figure: Message Handler.

Each time we store a message, we over-write its ID with value 240 (0xF0) to indicate that it is invalid. Previously we were using value zero, which is the timestamp channel, which resulted in the data clock being corrupted when we stored invalid messages by mistake. We introduce a delay after initiating serial transmission to the display panel, so that we will not attempt another such transmission before the first is complete. We set up a table that maps detector module index 0-15 to antenna number.

[07-DEC-22] We add a FIFO to handle outgoing messages to the display panel. Top antenna detection appears flawless. We place two dual-channel transmitters on separate antennas 30 cm apart, connected to DM6 and DM8. They are an A3028A2 No119 and A3028D3 No11. We record for ten minutes and observe the pattern of collisions between the two. In the ideal case, with the antennas far apart, we would expect no collisions because the detector modules are independent.

Figure: Collisions for Transmitters on Independent Antannas. The antennas are 30 cm apart. Recording M1670429697.ndf.

We place both transmitters on the same antenna and repeat. We are pleased to see that collisions are far less common.

Figure: Collisions for Transmitters on Same Antanna. Recording M1670431057.ndf.

We add a FIFO to handle incoming messages from the display panel. Reception of display panel messages is reliable and takes place between one and twenty microseconds after they arrive. We look at the daisy chain data strobe D1DS as it comes out of the controller logic and compare it to D16DS where it enters the sixteenth detector module. The data strobe is now propagaging through the detector modules as a combinatorial signal. The delay through fifteen modules is 150 ns, or 10 ns per module. A single message readout from DM16 is shown below.

Figure: Message Readout from DM16 with Ten Wait States Per Data Strobe Edge. Yellow: D1DS. Red: D16DS. Blue: DMRC (DC1). Green: MRDY (DC3).

right now we have 1.25-μs data strobe pulses. We are inserting ten CPU wait states after each data strobe transition. One wait state is 50 ns. We reduce the number of wait states to 5, which should reduce the length of the data strobe pulses by 250 ns. We look again at the readout, this time with an XOR of the data bits returning from the detector modules.

Figure: Message Readout from DM16 with Five Wait States Per Data Strobe Edge. Yellow: D1DS. Red: D16DS. Blue: DMRC (DC1). Green: XOR of data lines. Vertical: 1 V/div. Horizontal: 1 μs/div.

The entire readout of DM16 takes 10 μs. Each data strobe is 1 μs. The first transition of the data lines is provoked by the assertion of DMRC. The next follows the first assertion of data strobe, where DM16 presents the message ID to the controller. The glitch in the middle of the third data strobe occurs when the DM's Message Reader goes from state 6 to state 7. We do not yet understand the cause of the glitch. The round-trip delay of the daisy-chain, from the assertion of D1DS to data settling in the controller is 400 ns. The oscillations in data values at the end are an artifact of our addition of one to the upstream data through each detector module when calculating the antenna number.

Figure: Duplicate Message Readout. Yellow: D1DS. Red: D16DS. Blue: DMRC (DC1). Green: MRDY (DC3). Vertical: 1 V/div. Horizontal: 5 μs/div.

Here we see MRDY asserted because the three messages have been received by four detector modules. We read the first message then the second and store the first. We read the third and store the second. We read the fourth. After that we will store one or both of the third and fourth, but these actions are off the screen. The delay time between reads is 20 μs when we do not store and 25 μs when we do store. If we have ten 512 SPS transmitters in each of four FE3A, our total sample rate is 20 kSPS. If each sample is received four times, the time taken to handle each sample is 85 μs we can handle only 12 kSPS.

This photograph shows the routing of the antenna cables from detector modules to antenna inputs on the back wall of the TCB enclosure. The table below gives the mapping from detector module (DM) to antenna input (AI). The antenna index we obtain from reading a detector module through the daisy chain is the detector module number minus one. We combine this decrement with the above table to produce our mapping in the P3042BB code.

DM  AI  DM  AI
1   13  9   5
2   11  10  8
3   9   11  7
 4   6   12  10
 5   3   13  12
 6   4   14  14
 7   2   15  15
 8   1   16  16

This version of the firmware is too slow, but we finalize it as V3.2 and proceed with QC2 to test the antenna mapping, and we plan to overwhelm the TCB with excessive messages as a test of how well its performance degrades as it is overwhelms.

[08-DEC-22] We propose the following Quality Control Two (QC2) procedure. Configure the TCPIP interfaceRecord from TCB continuously with Neurorecorder. Play recording with Neuroplayer. Open Neurotracker window. Check Verbose, so we now see the top antenna power in the verbose output, as shown below. The "Tracker" line gives x, y, z position of the top antenna, as specified in the "alt" field of the NDF file's metadata. After that come sixteen values giving the detector module powers, all zero except for the top antenna.

Using 640 messages, including 128 clocks.
Channel 20,  0.0% loss, 512 reconstructed, 512 received, 0 bad, 0 missing.
Tracker: 48.0 8.0 2.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 178.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

In the lines above, the top antenna is number eight, assuming our mapping is correct from detector module to antenna input. The value "178" is the eight-bit digitized logarithmic power measurement provided by the top detector module. Connect metal-box test transmitter with center-frequency near 915 MHz via coaxial cable and attenuators to determine minimum input power for roughly 90% reception at each antenna input. Record this minimum along with the DM's own measurement of the same minimum power. Connect the same transmitter with no attenuator and record the maximum DM power measurement for saturated input. That's three numbers per detector module so far. Combine the signals from three metal-box test transmitters with center frequencies near 905 (LOW), 915 (MID), and 925 (HI) MHz using a four-way combiner with the fourth input terminated. Attenuate the combined signal so that power of each signal emerging is around −48 dBm, or some value that should be received by all TCB inputs. Apply this signal to all inputs and confirm that we see at least 90% reception from all three signals on each input. During this process, confirm that the detector module to antenna mapping is correct by watching the dots in the Neurotracker window, as shown below.

Figure: Neurotracker Window for the TCB. Antenna numbers are overlayed on this figure, not present in Neurotracker window.

For each antenna input and each of the three bands LOW, MID, and HI, record the signal power at which we tested and succeeded in getting reception. If we fail to see 90% reception or greater, increase the power until we do see reception, and record that power. That's seven numbers per antenna input total. record in a table similar to those we make for the ALT.

We re-work the Message Receiver in P3042DM, eliminating its readout glitches. Now we obtain the following when we read out detetor module DM16, with all sixteen DMs running P3042DM V2.3, the controller running P3042BB V3.2.

Figure: Message Readout of DM16 by P3042DM V2.3 and P3042BB. Yellow: D1DS. Red: D16DS. Blue: RDMSG on DM16 (TP3). Green: XOR of data lines at controller (DC3). Vertical: 1 V/div. Horizontal: 1 μs/div.

We eliminate some delays and drop the message readout time to 8 μs in a scratch version of P3042BB. The first byte is enabled onto the daisy chain with the rising edge of DS. The rest by the falling edge. The delay between the edge of DS and data ready at the controller 400 ns from DM16. Allowing 200 ns at the start to let the target module establish itself, 400 ns for each of five bytes read out, we could in principle reduce the readout time to 2.2 μs. We plan to add a state machine to P3042BB that reads out the daisy chain as fast as it can and writes the bytes into a buffer. The controller's CPU can read these bytes out as they come in, and write them to the relay message buffer. Our hope is to increase the message rate to 200 kMPS.

[09-DEC-22] Detector module buffer running well. Improve efficiency of main loop. With the interface running off a 2 MHz clock, we are reading messages and processing them in 6.5 μs. We arrange for four detector modules to receive the signal from a single transmitter, a dual-channel 512 SPS. Reception is reliable and robust. We see around 1450 MPS when we expect 1152 MPS. When the CPU interrupt disturbs duplicate elimination we end up saving extra messages, and our Receiver Instrument is not configured to purge duplicates from our signal, but the lwdaq_receiver routine always eliminates duplicates, giving preference to the most powerful in a sequence of duplicates from a TCB. In the figure below we see four detector modules being read out sequentially, and at the end the top antenna message being written by the CPU to the main message buffer.

Figure: Message Readout of Four Duplicates. Yellow: D1DS. Red: D16DS. Blue: DMRC (DC1). Green: Executing save_msg_prv. Red: Write to Detector Module Buffer. Vertical: 1 V/div. Horizontal: 2.5 μs/div.

The CPU handles messages in roughly 3 μs running at 20 MHz in the LCMXO2-7000HC5. The detector module readout time of 6.5 μs is the rate-limiting step. We double the clock speed of the Detetor Module Interface, but the readout is no longer reliable. We reduce the size of the RAM. The logic now fits in the 2000HC. The HC version of the chip runs off a 3.3-V power supply. It comes in speed grades 4-6, with 6 being the fastest. These speed grades support CPU clock speed 21 MHz up to 24 MHz. The HE versions run off 1.2-V core power, but support 3.3-V I/O. They come in the same speed grades 4-6 and support the same 21 MHz to 24 MHz CPU clock. The ZE versions run off 1.2-V core with 3.3-V I/O, but their speed grade is 1 only, for which maximum clock speed is 10 MHz. Even with this slower chip, we will still maintain an average 6.5 μs per message, so our message rate will be 154 kMPS. Consider twenty 512 SPS transmitters in one Faraday canopy with eight antennas and another twenty in a second canopy, all conneted to our TCB-A16. If we obtain 100% reception from all antennas within each canopy our message rate will be around 160 kMPS. We tag P3042BB V3.3. Later, while examining a recording made with many transmitters, we see duplicates remain prominent.

Messages 0 to 10 (index id value timestamp $hex):
    0   0 34688 123 $0087807B 0000
    1 136 39167   4 $8898FF04 390B
    2 133 41627   4 $85A29B04 790C
    3  11 57171  12 $0BDF530C A801
    4 153 39407  33 $9999EF21 5A02
    5  20 40887  37 $149FB725 640C
    6 135 39604  39 $879AB427 390B
    7 135 39604  39 $879AB427 640C
    8  12 57431  43 $0CE0572B A801
    9  12 57431  43 $0CE0572B 9C0C
   10 134 41286  46 $86A1462E 790A

Most samples are being received by four detector modules. Roughly half the samples are recorded twice. We examine the code and the occurrance of the double-messages. The message handler checks DMBRDY. If it's clear, it prepares to save the previous message. But first it checks the previous message is valid, checks that MRDY is not set, and checks that DMBBUSY is not set. An interrupt might occurr during these instructions. By this point, it could be that DMBRDY is set, and there is another byte available. In V3.3 we do not check for DMBRDY again. We will introduce a second check of DMBRDY and see if it eliminates the duplicates.

[11-DEC-22] The second check of DMBRDY eliminates almost all duplicates. We record for 400 s minutes in M1670683616.ndf. Here is a typical eleven-message sequence.

Messages 0 to 10 (index id value timestamp $hex):
    0   0  4096 123 $0010007B 0000
    1  12 39827  10 $0C9B930A 8F0E
    2 133 41534  11 $85A23E0B A40A
    3  20 39628  12 $149ACC0C 8004
    4 135 39235  24 $87994318 830A
    5  41 43266  25 $29A90219 900A
    6  11 40583  27 $0B9E871B 8F0E
    7 119 41608  40 $77A28828 8608
    8 136 39069  51 $88989D33 830A
    9  12 40105  53 $0C9CA935 900E
   10 134 41155  71 $86A0C347 A402

There are duplicates in the first eleven message of each archive we create. Here are the first few milliseconds of M1670683616.ndf.

Messages 0 to 10 (index id value timestamp $hex):
    0   0     0 123 $0000007B 0000
    1 133 41439   6 $85A1DF06 880D
    2 133 41439   8 $85A1DF08 A40A
    3 135 39274  13 $87996A0D 830A
    4 119 41850  18 $77A37A12 8608
    5  41 42847  25 $29A75F19 900A
    6  11 40084  38 $0B9C9426 8F10
    7 136 39103  52 $8898BF34 840A
    8  12 39457  52 $0C9A2134 9A0A
    9  12 39457  63 $0C9A213F 8F10
   10  20 39531  74 $149A6B4A 8004

There are occasional duplicates in the rest of the recording. Here is an example at time 8.0 s.

   46 135 39247  79 $87994F4F 830A
   47  11 40266  79 $0B9D4A4F 8F10
   48 133 41415  80 $85A1C750 880D
   49 133 41415 104 $85A1C768 A40A
   50 136 39103 104 $8898BF68 6A02
   51 136 39103 111 $8898BF6F 840A
   52  12 40488 124 $0C9E287C 8F0E

[12-DEC-22] We apply an SCT signal −8 dBm through attenuators to obtain a plot of detector module power measurement versus SCT signal power. The detector module is loaded onto its base board in a TCB enclosure. The signal enters through one of the enclosure antenna connections, propagates across a straight feedthrough, and then along a UMCC cable to the detector module.

Figure: Detector Module Power Calibration. Digitized, logarithmic power measurement (8-bit ADC counts) versus Input SCT signal power (dBm).

The plot stops at −62 dBm input, which is the minimum power at which the detector module can decode the incoming signal. Unlike the ALT, the TCB has no "background power" measurement for its antenna inputs, so we are unable to complete graphse like this.

[13-DEC-22] With one 512 SPS SCT signal fed into a single antenna input, we see perfect reception on the detector module but 50% reception by the base board. With our oscilloscope we observe that only half of messages are being stored. We find a bug in our CPU code where we have "jp z" instead of "jp nz". Correcting this error fixes our problem. We tag P3042BB 4V1, which we hope is ready for production.

[19-DEC-22] We arrange four antennas in an FE3A on a piece of wood, overlapping, so that we can place transmitter on all four at once and all four will carry their signals out of the enclosure. We split these four signals four ways each until we have sixteen antenna signals and feed them all into a single TCB-A16.

Figure: Piling Up Transmitters for TCB Overload Test.

According to the lights on the sixteen detector modules, all sixteen receive every message from every transmitter, barring those blocked by collisions. When we reached 13 kSPS total sample rate, the red DMERR light illuminates on the display panel, see here. The detector module message buffers are overflowing. Our message rate is now 208 kMPS. At 200 kMPS the DMERR light does not turn on. We estimated the maximum message rate would be around 154 kMPS, we are pleased to see the TCB performing well up to 200 kMPS. Once the buffers are full, the TCB continues to operate. Some message are being lost, but it appears we still have at least one copy of each message. So it is not clear that this form of failure will lead to loss of data.

[23-DEC-22] Nathan examines the extent to which one TCB detector module will detect the signal fed into a neighboring detector module, and in particular how the two shield covers over the preamplifier and limiting amplifier will affect such cross-talk. He reports, "The lowest power input required to experience crosstalk in the TCB is −17 dBm and this is true with or without the shields on the detector modules."

We have eight antennas in an FE5A canopy and 14 kSPS of transmitters each being received by most of the antennas. The antennas are split into sixteen signals and fed into a TCB-A16. We see DMERR flashing. In the current firmware v2.3, a detector module asserts DMERR for one of three reasons: write to full buffer, read from empty buffer, or failure of its phase-locked loop (PLL) to lock to the incoming DMCK. The first two conditions are latched, so the DMERR light on the display panel will remain lit until reset. The PLL error, however, persists only so long as the failure to lock persists. The flashing DMERR light on the display panel, which we observed also in our pervious, indicates intermittent failure of one or more detector module PLLs. We see the red light flashing on DM3 and later DM1 and DM2. The failure to lock stops when we turn off the transmitters and starts again when we turn them on again.

In the hours before we see the significance of the flashing DMERR light, we are examining the raw data from the TCB. We observe runs of duplicates spread over several clock ticks. In one case we observe the same message from the same detector module stored twice with different timestamps. We update detector module and base board firmware to delay or disable writing to all FIFO buffers when the buffers are full. Updating the base board firmware does not stop the duplicates. Our latest Receiver Instrument shows us raw messages and gives us a duplicate count we can watch as we are downloading the data. We are seeing 7k messages per half-second, containing 600 duplicates. These 7k are what is left of roughly 200k read from the detector modules after the base board's duplicate elimination. We re-program the detector modules starting at DM1 and working our way up to DM16. As we re-program a detector module, readout of the higher-numbered DMs is disabled and the duplicate count drops below 100. When we get to DM10, the duplicate count starts to rise. By the time we have reprogrammed all of them, the duplicate count is back to 600 and the red error lights are flickering on DM1 and DM2. We turn off the transmitters and the error lights turn off. With one or two transmitters in the enclosure, no detector modules fail to lock and duplicate elimination is perfect.

We suspect that activity on the daisy chain is inducing noise on DMCK. When the readout rate approaches 100 kMPS, we start to see failure to lock. The DMCK signal propagates from the base board logic chip past DM1 to DM16 and then to the display panel, where it reflects back to DM1. The DMCK signal is most corrupted by reflections at DM1 and DM2. It is these that fail to lock first. Right now we believe we have C48 = 100 pF loading DMCK at DM1 and 4 mA drive current from the base board logic chip.

[24-DEC-22] We remove C48. With 0 kSPS all DMs lock. At 5 kSPS DM1 fails to lock, and DM2-DM4 fail intermittently to lock.

 
Figure: Detector Module Clock. Scale: 1 V/div, 25 ns/div. Yellow: At DM1. Blue: at DM16. Left: C48 = 0 pF, 0 kSPS. Right: C48 = 0 pF, 5 kSPS.

We examine the clock at the start and end of a message readout using DMRC as our clock. We see no evidence that the readout is affecting the shape of DMCK. We let C48 = 100 pF and repeat. Once again, we see reliable lock with 0 kSPS, while at 5 kSPS DM1 always fails to lock and DM3 is intermittently locking.

 
Figure: Detector Module Clock. Scale: 1 V/div, 25 ns/div. Yellow: At DM1. Blue: at DM16. Left: C48 = 100 pF, 0 kSPS. Right: C48 = 100 pF, 5 kSPS.

At 5 kSPS we try 47 pF and 200 pF. In all cases we see reliable lock with 0 kSPS, while at 5 kSPS we have DM1 failing to lock.

 
Figure: Detector Module Clock. Scale: 1 V/div, 25 ns/div. Yellow: At DM1. Blue: at DM16. Left: C48 = 47 pF, 5 kSPS. Right: C48 = 200 pF, 5 kSPS.

The first few DMs will always lock if we remove any detector module up to and including DM10, but for DM11 and higher present, the lock is unreliable. We swap DM10 for a fresh DM and see the same behavior. We disconnect DM10's antenna and achieve the same effect as removing DM10. After ten minutes of making changes, we no longer see failure to lock in the first few DMs, but instead see buffer overflow in DM6, DM12, DM14-DM16. We have 5 kSPS entering all 16 antennas for total message rate of 80 kMPS. We record for 60 s. We see strings of duplicate messages in the recording. All DMs have new firmware that disables writing to the message buffer when the buffer is full.

We branch P3042DM and P3042BB to stream_configure branches in which we implement a configuration procedure following reset that allows the detector modules to determine their place in the daisy chain and remember it. The last DM in the chain is numnber one, and the first is number sixteen. Also branch P3042DM to quiet_read in which no ripple addition takes place, all DMs are number zero, so as to remove with minimal disruption the chaotic transisions caused by the asynchronous antenna number calculation on the daisy chain data bus.

[25-DEC-22] Reprogram all sixteen DMs with P3042DMQR1, quiet readout version one, in which we have no antenna number calculation and DM message buffers are only four messages deep. We place 5 kSPS in our Faraday canopy. We see no failure to lock. We see buffer overflow on DM16, DM15, and sometimes DM9-DM14. We reprogram DM12-DM16 with P3042DMQR2, in which detector module buffers are 32 deep. None of DM12-DM16 experience buffer overflow, but DM9-DM11 do overflow. No sign of flickering red lights on the detector modules, which implies no failure to lock. Continue until we have DM4-DM16 with buffers 32-deep, leaving DM1-DM3 4-deep. No buffer overflow. Increase sample rate in enclosure to 12 kSPS. Buffer overflow occurs immediately after reset, but we see no failure of detector modules to lock onto DMCK. Drop back to 5 kSPS. No buffer overflow, no failure to lock. The Receiver Instrument reports 4 kMPS downloaded, with 400 duplicates. Record 60 seconds in M1671988454.ndf. Leave the TCB running with the 5 kSPS to see if a overflow occurs over-night. Here is an example of duplication in the raw data, from the start of the recording.

   23  35 42092 205 $23A46CCD 640D
   24  12 39762 209 $0C9B52D1 A40D
   25  12 39762 210 $0C9B52D2 940D
   26  12 39762 210 $0C9B52D2 7A0D
   27  12 39762 210 $0C9B52D2 4B0D
   28  12 39762 211 $0C9B52D3 800D
   29  12 39762 211 $0C9B52D3 820D
   30  12 39762 212 $0C9B52D4 840D
   31  12 39762 212 $0C9B52D4 940D
   32  12 39762 213 $0C9B52D5 9D0D
   33  27 38830 231 $1B97AEE7 790D
   34  36 41759 239 $24A31FEF 630D

Here the antenna number is given always as 0x0D because the number reported by the DMs is zero, and the controller firmware maps number zero to TCB antenna input 0x0D = 13. We find a bug in the controller firmware, P3042BB. When the CPU reads the detector module buffer, we have been reading the input to the buffer (dmb_in) rather than the output (dmb_out). We correct and compile our master and self_configure branches of P3042BB.

[26-DEC-22] No DMERR on our TCB. Reprogram controller. No more consecutive duplicates in the raw data. Place a total of 11 kSPS in the Faraday canopy. The DMERR light turns on immediately. We suspect that some DMs may not have been re-programmed with the larger 32-deep buffer yesterday. We reprogram any that show an overflow. We record 60 s inM1672069013.ndf, no DMERR occurs. We appear to be reading roughly 150 kMPS. Our theoretical maximum message rate is 154 kMPS. Leave the TCB running overnight with 11 kSPS. We have several bugs in lwdaq_receiver, including one that crashes LWDAQ. We have disabled purging for our recording, and disabled duplicate replacement for TCB.

[27-DEC-22] No DMERR on TCB yet. Work on Receiver Instrument and Neurorecorder Tool to improve handling of errors. We purge duplicates in the Receiver Instrument for ODR and TCB, but not for ALT. We add to the Neuroplayer the option to purge duplicates from data it reads from an NDF file.

[04-JAN-23] Firmware P3042BB v4.2 provides a first effort at mapping the new daisy chain index to antenna socket on the box. Firmware P3042DM v3.1 implements configuration of daisy chain index after reset. We test both together. We see the antenna socket being calculated correctly from the index obtained during configuration. We see the index evolving as we reprogram detector modules and reset the TCB. The EMPTY and UPLOAD lights on the display panel are now behaving properly.

[24-JAN-23] We are exercising our six TCBs with reception tests, see here. We have three TCBs burned in. One we will keep here because of a problem with the enclosure, the other two are ready to ship.