PXIe-148X Getting Started Example - Troubleshooting Guide

Refer to this document to get insights and suggestions on troubleshooting performance issues, error codes, and problems with using the PXIe-148x Getting Started Examples (GSE).

Note: This document references the example included with the NI-FlexRIO 22Q3 driver. Examples included in newer releases of the driver should be applicable.

Table of contents

1. Error/Warnings Codes
2. Common Scenarios
3. Troubleshooting Guides
   1. How to debug incomplete I2C transaction errors (-304321)
   2. How to debug FIFO overflow errors (-52012)
   3. How to debug Generation GSE packet timing errors
   4. How to debug Acquisition FIFO overflow
   5. How to debug image display that does not show images or has a low frame rate
   6. How to debug issues with generation not starting
4. General Debugging Best Practices
   1. Host-FPGA General Debugging Workflow
   2. Get serial input channel status from the FPGA
   3. Get serial output channel status from the FPGA
   4. Instrumenting and Monitoring FPGA Behavior
      1. Add additional indicators that allow you to poll and view the information from the host.
      2. You can latch signals and add new indicators
      3. You can count rising and falling edges to see where things are going wrong
      4. Add an FPGA to Host FIFO to send up debug information sideband.
   5. Optimize System Settings for Streaming Performance
5. Related Documents

---

Error/Warnings Codes

Common Scenarios

In additional to observed error codes, some common scenarios can occur that require investigation.

Troubleshooting Guides

How to debug incomplete I2C transaction errors (-304321)

Most errors can be resolved with one of the following steps:

1. On a PXIe-1487 or PXIe-1489, there are three loopback scripts for odd channels, even channels, or both. Ensure the correct script is used for the desired serial channel.
2. A camera may not be powered on. Double check the Power over Coax settings are correct to power on the camera from the PXIe-148x acquisition board.
3. Script may have invalid commands - need to compare camera data sheet vs script commands

For help in determining the root cause of the error, wire the script output from the Run Configuration Script VI to an indicator and enable the Log I2C to Disk control. Run the VI and compare the script output to the logged I2C traffic.

FlexRIO Driver version 24Q1 Update - The SerDes Configuration Script Utility was updated to run as a TCP server with the host LabVIEW code operating as a TCP client to send commands for running SerDes configuration scripts. The update improves configuration script execution time when running more than one script, but also requires removal of the script output parameter in the Run Configuration Script VI. Troubleshooting is now done by running the stand-alone SerDes Configuration Script Example. See the SerDes Configuration Script Tutorial for details.

How to debug FIFO overflow errors (-52012)

The Getting Started Examples use many FIFOs to pass data and metadata between the Host and the FPGA. While many of these FIFOs are small or transmit small amounts of data, any of them can fail as you approach your total system bandwidth limitations. Below are the FIFOs you can find the in the Acquisition, Tap, and Generation examples and links to other debugging steps to take when encountering these errors.

• Acquisition/ TAP FIFOs
  • LLP Packets FIFO - There is one of these FIFOs per serial channel. These can overflow if the FIFO sending data from the FPGA to the host does not have any space in it. This can happen when we do not write LLP packets to disk on the host side fast enough and make space in the FIFO for incoming data. For a more detailed explanation refer to How to debug Acquisition FIFO overflow.
• Common FIFOs - These FIFOs are small and do not move much data to the Host. If one of these FIFOs overflows, it usually indicates a system bandwidth limitation. Try following steps from other troubleshooting guides below.
  • Image Data FIFO - There are two of these FPGA to Host FIFOs by default in each GSE. Refer to How to debug image display that does not show images or has a low frame rate.
  • Image Metadata FIFO - There are two of these FPGA to Host FIFOs by default in each GSE. Refer to How to debug image display that does not show images or has a low frame rate.
  • I2C Timestamp FIFO - There is one of these FPGA to Host FIFOs by default in each GSE.
  • GPIO Timestamp FIFO - There is one of these FPGA to Host FIFOs by default in each GSE.
• Generation FIFOs - It should not be possible to see a FIFO overflow on the Host to FPGA LLP Packets FIFOs. The generation GSE holds off on writing data to the FIFO until the FIFO has space available. Refer to How to debug Generation GSE packet timing errors for more Generation GSE specific debugging.

How to debug Generation GSE packet timing errors

The first thing to check is the Serial Output Channel Status indicator. Verify that the bytes going in and coming out of the DRAM match. If the DRAM manager is overflowing, it means the number of packets being sent out the serial output channel is less than then packets being sent from the host. Verify the TDMS data set has timestamps running at the correct rate. Refer to Get serial output channel status from the fpga.

If you cannot figure out what part of the serial output data path is causing the timing errors, or are seeing FIFO overflow errors, take these steps to reduce system bandwidth. Refer to Reducing system bandwidth usage. Afterwards, try some of these tasks to see which part of your system or data set is responsible for causing overflows.

• Disable display (or shrink image size)
• Use smaller packets (reduce bandwidth requirements)
• Set your timing error limit so high that the system does not error out, and then see if DRAM manager still fills up
• Use the TDMS file creator to create a known good set of CSI-2 packet data with the desired timing characteristics and verify your generation completes successfully
• Reduce serial output channel count

How to debug Acquisition FIFO overflow

The first thing to check is the Serial Input Channel Status indicator. You should verify that the bytes going in and coming out of the DRAM match. If the DRAM manager is overflowing, it likely means the FPGA to Host FIFO is pushing back on the DRAM manager and the DRAM partition is getting full. Refer to Get serial input channel status from the FPGA.

It is hard to distinguish if DMA is pushing back on DRAM causing DRAM overflow (e.g. system can’t receive data fast enough, this can be many root causes), or if it’s just data coming in faster than we can handle (FPGA simply can’t handle this data rate).

• If downstream is not ready for output and you overflow, likely it’s downstream at issue (DMA pushing back)
• If downstream is ready, and you overflow, DRAM manager might not be able to handle. This scenario is generally unlikely based on empirical testing.

If one of the two scenarios above is suspected, you can instrument the FPGA to determine where in the data path you are getting clogged up. Refer to Instrumenting and Monitoring FPGA Behavior.

Things we can suggest to look into:

• Double check Active Lane rate and count are matched throughout your system (Camera -> Instrument)
• Look at the Serial Channel Input Status array for patterns, are problems always following a specific channel?
• Verify DRAM filling, try to compare rate of DRAM filling vs FIFO transfer rate to host
• Reduce the acquisition to a single channel and verify that it stops successfully.

How to debug image display that does not show images or has a low frame rate

Images display with a reduced frame rate or do not display at all when the system does not have the bandwidth to keep up with the images coming from the FPGA.

Refer to Reducing system bandwidth usage to reduce the bandwidth display is using.

How to debug issues with generation not starting

If generation never starts, the first thing to check is the generation start thresholds configured in the host VI Configure Serial Output Channels.vi.

• timestamp buffer start threshold - The minimum number of timestamps that must be buffered before generation begins
• packet data buffer start threshold (bytes) - The minimum number of valid CSI-2 packet bytes that must be buffered before generation begins
• dram partition start threshold (bytes) - The number of bytes that dram partition must have stored before generatoin begins

The most common way to force the generation to start is the reduce the above thresholds to zero, and increase the packet timing error threshold (cycles) to a very large number (e.g. FFFFFFFFFFFFFFF0) so that it will never have a timing error and generation will start immediately. Then you can look at the Serial Channel Output Status indicator to determine if the channel is otherwise behaving correctly.

If modifying the thresholds on the host side is not giving you sufficient insight, you can instrument the FPGA VI Monitor Ready For Start Condition.vi to see which of the three threshold conditions is not being met. Refer to Instrumenting and Monitoring FPGA Behavior.

General Debugging Best Practices

Host-FPGA General Debugging Workflow

The overall architecture of our Host and FPGA example code is to have the host query and poll at appropriate times to pass the status of the FPGA to the user. Errors that are detected at runtime on the FPGA are bubbled up through boolean status signals on the FPGA front panel. The host will read these signals and set an error message so that the source of the error can be found. Errors that are set will tell the user where to start looking for errors. Looking at the host code will tell you which portions of the FPGA design to look at. The signals are latched until the host api calls reset or FPGA diagram reset.

Additionally the host will check certain statuses that can give you clues about how or why an error occurred. The Acquisition and Generation states are queried and will display “Error” if something has gone wrong.

There are other times where we wait for a certain state to be returned from the FPGA. If a host side timeout occurs while waiting for that state, we will set an error.

Get serial input channel status from the FPGA

One of the best ways to diagnose Acquisition or TAP FPGA errors is to monitor the Serial Input Channel Status indicator from the FPGA. This indicator can be added to the Acquisition or Tap GSE from the Read Serial Input Channel Status VI.

The Serial Input Channel Status is an array of status cluster indicators - one for each channel. The cluster has the following indicators:

• acq state - Indicates the current acquisition state of the channel.
• stop timed out - The last packet header was not detected within Stop Timeout (s) seconds.
• bad packet header - This indicator is set if there was an uncorrectable header error.
• bad packet payload - This indicator is set if the CRC of the payload was incorrect.
• bad packet size - This indicator is set if the packet was truncated and did not match the size in the header.
• dram status - This indicator gives you insight into the state of the DRAM manager.
  • bytes written - Number of bytes written to this channel’s DRAM partition.
  • bytes read - Number of bytes read from this channel’s DRAM partition.
  • overflow - The DRAM partition could not be written to because the partition was full.
  • partition full - A live status showing when the partition is full. The partition can be full, it just needs to be read from before additional writes are made.
  • last byte flushed - An end of acquisition status where the last valid byte of the last packet has been read from the DRAM partition.

What kinds of scenarios can you sniff out based on the indicator behavior?

• If acq state is stuck in idle, we are never detecting a valid CSI-2 packet.
• If stop timed out is true, we did not receive the last packet in the expected amount of time. Increase the Stop Timeout (s) control on the front panel to see if the packet is late, or if there is a missing packet.
• If any of the bad packet indicators are lit, we are not receiving valid CSI-2 packets. Check your camera and connection to the serial input channel.
• If the acquisition has ended but bytes written is greater than bytes read then there is valid data getting stuck in the DRAM partition and something downstream is not ready to receive CSI-2 packets.
• If the overflow indicator is true, the DRAM manager and data path is not keeping up with the incoming packets on the serial input channel. This can be caused from the FPGA to Host FIFO backing up and pushing back on the DRAM manager. This can happen if the data is not being written to disk fast enough on the host or if data is not being pulled out of the serial channel FIFO fast enough on the host side.
• If the partition full indicator is true, the partition needs to be read from. You can instrument this signal on the FPGA side to see how often the partition is getting full. It likely bounces between full and not full many times before finally overflowing. This is indicative of the host FIFO or data path not being able to keep up with incoming data on the serial input channel.
• If the last byte flushed stays false when an acquisition completes, the last packet will be missing from the data set.

Get serial output channel status from the FPGA

One of the best ways to diagnose Generation errors is to monitor the Serial Output Channel Status indicator from the FPGA. This indicator can be added to the Generation GSE from the Read Serial Output Channel Status VI.

The Serial Output Channel Status is an array of status cluster indicators - one for each channel. The cluster has the following indicators:

• num buffered llp packets - Indicates the number of LLP packets that are waiting to be generated.
• packet timing error (cycles) - Indicates how many cycles the timestamp of the generated packet is behind expected time to generate.
• packet timing error - This indicator is set if the timestamp of the generated packet was beyond the allowed packet timing error of the data.
• ready to generate packet - This indicator toggles true every time the downstream data path is ready to accept a CSI-2 packet.
• dram status - This indicator gives you insight into the state of the DRAM manager.
  • bytes written - Number of bytes written to this channel’s DRAM partition.
  • bytes read - Number of bytes read from this channel’s DRAM partition.
  • overflow - The DRAM partition could not be written to because the partition was full.
  • partition full - A live status showing when the partition is full. The partition can be full, it just needs to be read from before additional writes are made.
  • last byte flushed - An end of acquisition status where the last valid byte of the last packet has been read from the DRAM partition.
• num buffered llp timestamps - The number of LLP timestamps that are waiting to be generated. This should be close to the number of buffered LLP packets.
• generation done - Indicates the generation has completed.
• num generated packets - The number of CSI-2 packets that were sent out the serial output channel.

What kinds of scenarios can you sniff out based on the indicator behavior?

• If the generation has ended but bytes written is greater than bytes read then there is valid data getting stuck in the DRAM partition and something downstream is not ready to receive CSI-2 packets.
• If the overflow indicator is true, the DRAM manager and data path is not keeping up with the incoming packets on the serial output channel. This can be caused if the host is sending more data than the DRAM manager or serial output data path can process.
• If the partition full indicator is true, the partition needs to be read from. You can instrument this signal on the FPGA side to see how often the partition is getting full. It likely bounces between full and not full many times before finally overflowing. This is indicative of the host FIFO or data path not being able to generate data fast enough on the serial output channel.
• If the last byte flushed stays false when a generation completes, the last packet will never have been generated.
• If the number of generated packets does not match the expected number of generated packets, the data set might not be valid. This can happen if the data is expected to run faster than the serializer can output, or if the packets are getting stuck in DRAM.

Instrumenting and Monitoring FPGA Behavior

When debugging or investigating FPGA behavior, there are some general strategies that you can implement to figure out where errors are coming from. You can monitor the boolean signals that are passed to the host through the FPGA front panel, or you can monitor various internal state machines. The exact nature of what you should instrument is dependent on the problem you are trying to solve, but there are three common strategies for getting debug information out of the LabVIEW FPGA image at runtime. > Note: All of the strategies below will require the FPGA to be recompiled to reflect the suggested changes.

Add additional indicators that allow you to poll and view the information from the host.

Adding indicators will allow you to read the status back of interesting signals from the host. This will give you a “last updated” view of the signals in question. It will not allow you to monitor every change to a signal but can give you good insights.

You can latch signals and add new indicators

Dropping down a basic latch when a certain condition occurs can allow you to save a particular state. Make sure to wire up the reset signal so that you can clear your state. This will allow you to capture a specific change on the FPGA.

You can count rising and falling edges to see where things are going wrong

This is a good way to measure boolean signals to see how many times a certain condition occurs. This can give you insights into complex data flow problems wondering how many times a gate or state is allowing valid data to flow.

Add an FPGA to Host FIFO to send up debug information sideband.

This technique takes a long time to setup, can dramatically change the resource utilization, and can change overall system bandwidth usage. It is the best way to instrument every single change on a certain signal and pipe it up to the host. Having the host pull data from the fifo and store it can lead to bandwidth and file system concerns, so only use this technique if you really need to capture every change on a signal in the FPGA.

Optimize System Settings for Streaming Performance

Section Coming Soon

---

Related Documents

• PXIe-148X Getting Started Example - Generation Help
• PXIe-148X Getting Started Example - Acquisition Help
• PXIe-148X Getting Started Example - Tap Help