Troubleshooting Storage Bottlenecks & USB Disconnects on Jetson AGX Xavier

When preparing an edge device for large-scale AI dataset management and model fine-tuning, storage stability is paramount. During a recent setup on an NVIDIA Jetson AGX Xavier, we encountered an unexpected storage failure mode that shifted our focus from simple filesystem configuration to kernel-level hardware debugging.
Here is a breakdown of our objective, the issues faced, the root-cause analysis, and how we solved it.

1. What Was Our Objective?Our primary goal was to configure an external 3.6 TB SATA HDD (connected via a USB-to-SATA bridge) on a Jetson AGX Xavier running Ubuntu 20.04.
   The Goal: Establish a dedicated, high-capacity workspace for housing training image datasets, annotations, and export pipelines.
   The Plan:

• Mount the 3.6 TB partition (/dev/sda24) to /mnt/finetune_lprnet.
• Configure /etc/fstab for auto-mounting on reboot.
• Set up directory structures for datasets (train, val, test, annotations).

2. What Issue Did We Encounter? Everything appeared fine initially when inspecting df -h and lsblk.

• However, as soon as we attempted to create project directories using mkdir -p, the system began throwing errors:

mkdir: cannot create directory ‘/mnt/finetune_lprnet/lpr_project’: Read-only file system

• When trying to remount and re-execute, the error escalated:

mkdir: cannot create directory ‘/mnt/finetune_lprnet/lpr_project’: Input/output error

Escalation Breakdown:Read-only file system: The Linux kernel encountered an unexpected write failure and remounted the ext4 filesystem as Read-Only to prevent data corruption. Input/output error: The block layer completely lost communication with the physical drive.

3. How Did We Solve It? Instead of blindly running fsck repair commands—which can worsen corruption if the physical link is failing—we inspected kernel log outputs (dmesg) to identify the actual culprit.
   Root Cause Analysis via Kernel Logs: Inside dmesg, we spotted the following critical log sequence:

[ 269.130514] usb 2-1: USB disconnect, device number 2
[ 269.131126] blk_update_request: I/O error, dev sda, sector 0 op 0x1:(WRITE) flags 0x800...
[ 269.714810] usb 2-1: new SuperSpeed Gen 1 USB device number 3 using tegra-xusb
[ 269.735806] Manufacturer: JMicron
[ 269.744661] scsi host1: uas

Key Takeaways from the Logs:

• USB Bus Disconnect: The JMicron USB-to-SATA bridge physically dropped off the USB bus during write operations (USB disconnect, device number 2) and re-enumerated as a new device (device number 3).
• Protocol Instability: The drive was utilizing UAS (USB Attached SCSI). Certain USB bridge chips (like JMicron JMS578/JMS567) exhibit unstable UAS performance or power-draw spikes under load on embedded Linux kernels.
  Step-by-Step Resolution:
• Unmount the corrupted mountpoint safely:

sudo umount -l /mnt/finetune_lprnet

• Perform Filesystem Repair (fsck): Once the device stabilized, we repaired journal errors caused by the abrupt unmount:

sudo fsck.ext4 -y /dev/sda24

• Configure Persistent & Safe Mounting (fstab): Updated /etc/fstab with nofail so that unexpected drive disconnects wouldn't freeze system boots into Emergency Mode:

UUID=4c2e282c-bb97-4b27-9e33-5624c8d4a6b4 /mnt/finetune_lprnet ext4 defaults,nofail 0 2

Hardware & Driver Mitigation: Power Supply: Ensured the 3.5" HDD was powered via its dedicated power adapter rather than drawing bus power from Jetson's USB ports.
UAS Quirks Workaround (If disconnects persist): Applied a kernel quirk (usb-storage.quirks=152d:0578:u) to force the standard usb-storage driver instead of uas, ensuring much higher tolerance to power dips and link drops.

4. What Succeeded?After fixing the filesystem errors and securing the hardware connection, we successfully remounted the drive and verified write capabilities:

# Mount and reclaim permissions
sudo mount -a
sudo chown -R $USER:$USER /mnt/finetune_lprnet

# Successfully created nested dataset directory structures!
mkdir -p /mnt/finetune_lprnet/lpr_project/{raw_images,annotations,transformed,datasets/{train,val,test}/{images,labels},exports}

We also established a symbolic link to the home directory for easy project access:

ln -s /mnt/finetune_lprnet/lpr_project ~/lpr_project

---

💡 Key Lessons for Edge AI Developers:

• Don't ignore early Read-only warnings: They are safety mechanisms triggering before total I/O failure.
• Check dmesg before running repair tools: Always verify whether an error is filesystem logic corruption or a physical USB drop.
• Watch out for USB-to-SATA bridge chips: Embedded Linux boards often need powered hubs or UAS quirks to keep heavy external storage devices stable.

#EmbeddedSystems #Linux #NVIDIA #JetsonAGX #EdgeAI #Storage #Troubleshooting #DevOps #DeepLearning