External HDD turns read-only

Troubleshooting
1

I have a 14TB Seagate Exos hard drive that out of nowhere began exhibiting read-only errors. I use it as a NFS share and Plex media storage. I have tried unmounting and mounting it using dietpi-drive_manager. Initially it shows as rw but quickly reverts to read-only. I have tried moving the dietpi_userdata location to and from the hard drive.

Here are the errors


dietpi-drive_manager error when trying to mount rw
[FAILED] Could not apply                                           
                                                                   
Error log:                                                         
 - mount: /mnt/Seagate-14TB: mount point not mounted or bad option.
       dmesg(1) may have more information after failed mount system

cp: preserving times for '/mnt/Seagate-14TB/dietpi_userdata/.': Read-only file system
[FAILED] DietPi-Set_userdata | Failed to copy /mnt/dietpi_userdata/ to /mnt/Seagate-14TB/dietpi_userdata.
[FAILED] DietPi-Set_userdata | Exited with error

dietpi@DietPi:~$ sudo dmesg
[  345.948972] sd 1:0:0:0: [sdb] tag#12 CDB: Read(16) 88 00 00 00 00 00 0c 81 2f c8 00 00 00 08 00 00
[  345.948980] critical target error, dev sdb, sector 209792968 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
[  345.949020] BTRFS error (device sdb1: state EA): bdev /dev/sdb1 errs: wr 9, rd 2242, flush 0, corrupt 11831, gen 0
[  358.203709] sd 1:0:0:0: [sdb] tag#13 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=2s
[  358.203767] sd 1:0:0:0: [sdb] tag#13 Sense Key : Medium Error [current] 
[  358.203779] sd 1:0:0:0: [sdb] tag#13 Add. Sense: Unrecovered read error
[  358.203790] sd 1:0:0:0: [sdb] tag#13 CDB: Read(16) 88 00 00 00 00 00 0c 81 2f c8 00 00 00 08 00 00

dietpi@DietPi:~$ sudo btrfs check /dev/sdb1
Opening filesystem to check...
Checking filesystem on /dev/sdb1
UUID: 61245d2d-8a69-44df-b612-6f28d9d35d98
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 7842252328960 bytes used, no error found
total csum bytes: 7648136500
total tree bytes: 8567734272
total fs tree bytes: 271532032
total extent tree bytes: 179159040
btree space waste bytes: 436071653
file data blocks allocated: 7833684594688
 referenced 7833675567104

dietpi@DietPi:~$ sudo dietpi-software reinstall 114
│  - Command: chown dietpi:dietpi /mnt/dietpi_userdata/Music          │
│ /mnt/dietpi_userdata/Pictures /mnt/dietpi_userdata/Video            │
│ /mnt/dietpi_userdata/downloads                                      │
│  - Exit code: 1                                                     │
│  - DietPi version: v9.2.1 (MichaIng/master) | HW_MODEL: 21 |        │
│ HW_ARCH: 10 | DISTRO: 7                                             │
│  - Image creator: DietPi Core Team                                  │
│  - Pre-image: Debian Buster                                         │
│  - Error log:                                                       │
│ chown: changing ownership of '/mnt/dietpi_userdata/Music':          │
│ Read-only file system                                               │
│ chown: changing ownership of '/mnt/dietpi_userdata/Pictures':       │
│ Read-only file system                                               │
│ chown: changing ownership of '/mnt/dietpi_userdata/Video':          │
│ Read-only file system                                               │
│ chown: changing ownership of '/mnt/dietpi_userdata/downloads':      │
│ Read-only file system ```
2

Does the drive has it’s own power supply or is it just powered via the USB ports?
You can check dmesg for under-voltage errors:

dmesg | grep Under-voltage

If you already use a powered USB hub or something similiar I would say you should make a backup of the drive, it’s maybe dying.

3

let’s check for kernel errors in general.

dmesg -l 0,1,2,3

@aw512 due to whatever reason you ignored the troubleshooting template. Pls fill the required information

Required

  • DietPi version | cat /boot/dietpi/.version
  • Distro version | echo $G_DISTRO_NAME $G_RASPBIAN
  • Kernel version | uname -a
  • Architecture | dpkg --print-architecture
  • SBC model | echo $G_HW_MODEL_NAME or (EG: RPi3)
4

It is a Seagate Backup Plus Hub which has its own external power supply.

dmesg | grep Under-voltage returns nothing.

root@DietPi:~# dmesg -l 0,1,2,3
[   36.699829] critical target error, dev sdb, sector 22336040 op 0x0:(READ) flags 0x80700 phys_seg 64 prio class 2
[   36.699928] critical target error, dev sdb, sector 947683872 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 2
[   36.699950] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 10, rd 2234, flush 0, corrupt 11831, gen 0
[   36.700008] critical target error, dev sdb, sector 209792968 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 2
[   36.700098] critical target error, dev sdb, sector 383072 op 0x1:(WRITE) flags 0x1800 phys_seg 8 prio class 2
[   36.700117] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 11, rd 2234, flush 0, corrupt 11831, gen 0
[   36.700137] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 12, rd 2234, flush 0, corrupt 11831, gen 0
[   36.700191] critical target error, dev sdb, sector 2480224 op 0x1:(WRITE) flags 0x1800 phys_seg 8 prio class 2
[   36.700209] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 13, rd 2234, flush 0, corrupt 11831, gen 0
[   36.700256] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 14, rd 2234, flush 0, corrupt 11831, gen 0
[   36.848599] DMAR: DRHD: handling fault status reg 3
[   36.848614] DMAR: [DMA Write NO_PASID] Request device [00:15.0] fault addr 0xff374000 [fault reason 0x05] PTE Write access is not set
[   36.848627] DMAR: DRHD: handling fault status reg 3
[   36.848635] DMAR: [DMA Write NO_PASID] Request device [00:15.0] fault addr 0xff376000 [fault reason 0x05] PTE Write access is not set
[   36.848682] DMAR: DRHD: handling fault status reg 3
[   36.848696] DMAR: [DMA Write NO_PASID] Request device [00:15.0] fault addr 0xfe942000 [fault reason 0x05] PTE Write access is not set
[   36.848712] DMAR: DRHD: handling fault status reg 3
[   36.851996] xhci_hcd 0000:00:15.0: ERROR Transfer event TRB DMA ptr not part of current TD ep_index 2 comp_code 1
[   39.565365] critical target error, dev sdb, sector 209792968 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
...
[  738.635239] BTRFS error (device sdb1: state EMA): Remounting read-write after error is not allowed

Here is the requested template info

dietpi@DietPi:~$ cat /boot/dietpi/.version
G_DIETPI_VERSION_CORE=9
G_DIETPI_VERSION_SUB=2
G_DIETPI_VERSION_RC=1
G_GITBRANCH='master'
G_GITOWNER='MichaIng'
dietpi@DietPi:~$ echo $G_DISTRO_NAME
bookworm
dietpi@DietPi:~$ uname -a
Linux DietPi 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64 GNU/Linux
dietpi@DietPi:~$ dpkg --print-architecture
amd64
dietpi@DietPi:~$ echo $G_HW_MODEL_NAME
Native PC (x86_64)
5

Probably some corruption on your HDD. Can you try to umnount the HDD abd check following

fsck -af /dev/
6

Prior to posting I did try fsck without -af, but it says for BTRFS filesystems to use btrfs check. Which I did and the results are in the OP.

root@DietPi:~# fsck -af /dev/sdb1
fsck from util-linux 2.38.1
root@DietPi:~# fsck /dev/sdb1
fsck from util-linux 2.38.1
If you wish to check the consistency of a BTRFS filesystem or
repair a damaged filesystem, see btrfs(8) subcommand 'check'.


│  - User data location: RootFS (/mnt/dietpi_userdata)                                                                 │              
│                                                                                                                      │              
│                                                   ●─ sda ────────────────────────────────────────────────            │              
│         /                                         : /dev/sda2 | ext4 | Capacity: 117.4G | Used: 21.8G (19%)          │              
│         /boot/efi                                 : /dev/sda1 | vfat | Capacity: 92.5M | Used: 16.7M (18%)           │              
│                                                   ●─ sdb ────────────────────────────────────────────────            │              
│         /mnt/61245d2d-8a69-44df-b612-6f28d9d35d98 : /dev/sdb1 | btrfs | Not mounted                                  │              
│                                                   ●─ mmcblk0 ─────────────────────────────────────────────●          │              
│         /mnt/c057a1b1-ade3-4a80-8108-dcc4f55d57b7 : /dev/mmcblk0p1 | btrfs | Not mounted
7

Your command needs to be for an unmounted partition
sudo btrfs check --progress /dev/sdb1
it might take a while…
if damaged, a repair (from below cheatsheet) might help

This is a good BTRFS cheatsheet

make sure you have 100% stable power to the external drive…USB powered is NOT nearly stable enough

Might also look into doing a BTRFS scrub as well…set a crontab to run it monthly just to ensure the data/partition is kept clean

Also…might look to see if maybe the USB has sleep turned on…if the USB ports on the device go to sleep it might corrupt the data read/writes to the device
There is also a bunch of really nice scripts written for BTRFS maintenance

8

The oem power supply is 12 Volts 3 Amps, I would think power would not be an issue.

Thank you for the btrfs cheatsheet. It has a lot of troubleshooting steps I can run. Looking into btrfs troubleshooting also led me to this guide. Currently I am running

btrfs scrub start -Bd /mnt/Seagate-14TB

It worked for someone else with the same issue.

Every 1.0s: sudo btrfs scrub status /mnt/Seagate-14TB                                               DietPi: Tue Apr  9 19:53:07 2024

UUID:             61245d2d-8a69-44df-b612-6f28d9d35d98
Scrub started:    Tue Apr  9 19:22:24 2024
Status:           running
Duration:         0:30:43
Time left:        610:51:46
ETA:              Sun May  5 06:44:53 2024
Total to scrub:   7.14TiB
Bytes scrubbed:   6.12GiB  (0.08%)
Rate:             3.40MiB/s
Error summary:    read=3
  Corrected:      0
  Uncorrectable:  3
  Unverified:     0

The HDD is plugged in to a USB 3.0 port. I am not sure why the rates are so slow. The scrub ETA is over 3 weeks :skull:

I have never ran any BTRFS maintenance. I need to implement the linked scripts.

9

The scrub would abort less than 15 minutes in on the DietPi host. I removed the hard drive, scrubbed and defragmented it on as NixOS laptop. On the laptop the rw issue and dmesg errors mentioned prior were not present.

UUID:             61245d2d-8a69-44df-b612-6f28d9d35d98
Scrub started:    Tue Apr  9 21:36:53 2024
Status:           finished
Duration:         9:30:14
Total to scrub:   7.14TiB
Rate:             218.71MiB/s
Error summary:    read=5
  Corrected:      0
  Uncorrectable:  5
  Unverified:     0

Afterwards, I plugged the HDD to the DietPi system, and used dietpi-drive_manager to mount the drive to the same /mnt/Seagate-14TB mount point.
Similar dmesg errors occured.

[ 28.532338 ] critical target error, dev sdb, sector 209792968 op exe: (READ) flags 0x80700 phys_seg 4 prio class 2 31.298102] critical medium error, dev sdb, sector 209792968 op exe: (READ) flags exe phys seg 1 prio class 2 class 2
[ 34.930923 ] critical target error, dev sdb, sector 209792968 op 0x0: (READ) flags exe phys seg 1 prio 34.931036] BTRFS error (device sdbi): bdev/dev/sdbi errs: wr 15, rd 2588, flush e, corrupt 17658, gen
[ 41.661914 ] critical target error, dev sdb, sector 209792968 op 0x0: (READ) flags exe phys seg 1 prio class 2 41.661962] BTRFS error (device sdbi): bdev/dev/sdbi errs: ur 15, rd 2589, flush 0, corrupt 17658, gen sector 209792968 op 0x0: (READ) flags exe phys seg 1 prio class 2
[ 51.675838 ] critical target error, dev sdb,
[ 51.675900 ] BTRFS error (device sdbi): bdev/dev/sdb1 errs: ur 15, rd 2590, flush 8, corrupt 17658, gen 0 63.881053) critical target error, dev sdb, sector 27150848 op 0x0: (READ) flags 0x80708 phys seg 64 prio class 2
[ 63.881130 ] critical target error, dev sdb, sector 25295872 op 0x0: (READ) flags 0x80700 phys_seg 64 prio class 2 63.881175] critical target error, dev sdb, sector 2736264288 op 0x0: (READ) flags 0x1000 phys seg 4 prio class 2
[ 63.881210 ] BTRFS error (device sdbi): bdev/dev/sdb1 errs: ur 15, rd 2591, flush e, corrupt 17658, gen e
[ 63.881282 ] critical target error, dev sdb, sector 209792968 op 0x0: (READ) flags exe phys seg 1 prio class 2
[ 63.881314 ] BTRFS error (device sdbi): bdev/dev/sdb1 errs: ur 15, rd 2592, flush 8, corrupt 17658, gen @
[ 64.007258 ] DMAR: DRHD: handling fault status reg 3
[ 64.007296 ] DHAR: [DMA Write NO_PASID] Request device [00:15.0] fault addr 0xfe980000 [fault reason 0x05] PTE Write access is not set
[ 76.111497 ] critical target error, dev sdb, sector 1135877632 op 0x0: (READ) flags 0x80700 phys_seg 64 prio class 2
[ 76.111710 ] critical target error, dev sdb, sector 209792968 op 0x0: (READ) flags exe phys seg 1 prio class 2 ] BTRFS error (device sdb1): bdev/dev/sdb1 errs: ur 15, rd 2593, flush 8, corrupt 17658, gen e
[ 76.111932 ] critical target error, dev sdb, sector 1101075352 op 0x0: (READ) flags 0x80700 phys_seg 64 prio class 2 76.112051] critical target error, dev sdb, sector 25381888 op exe: (READ) flags 0x80700 phys_seg 64 prio class 2
[ 76.112176 ] critical target error, dev sdb, sector 6116648 op 0x0: (READ) flags 0x80700 phys_seg 64 prio class 2
[ 76.345318 ] DMAR: DRHD: handling fault status reg 3 DHAR: [DMA Write NO_PASID] Request device [00:15.0] fault addr 0xfe980000 [fault reason 0x05] PTE Write access is not set
[ 76.345355 ] DMAR: [DMA Write NO PASID] Request device [00:15.0] fault addr exfec40000 [fault reason 0x05] PTE Hrite access is not set
[ 88.774243 ] critical target error, dev sdb, sector 209792968 op 0x0: ( READ) flags exe phys seg 1 prio class 2
[ 88.774295 ] BTRFS error (device sdb1): bdev/dev/sdbi errs: ur 15, rd 2594, flush 0, corrupt 17658, gen e dev sdb, sector 25757184 op 0x0: (READ) flags 0x80700 phys_seg 64 prio class 2
[ 88.774342 ] critical target error,
[ 88.774388 ] critical target error, dev sdb, sector 5759440 op 0x0: (READ) flags 0x80700 phys_seg 64 prio class 2 ] DMAR: DRHD: handling fault status reg 3
[ 89.044351 ] DMAR: [DMA Write NO PASID] Request device [00:15.0] fault addr exfecc0000 [fault reason 0x05] PTE Hrite access is not set

Mounting the drive to a different mount point did not result in the dmesg errors, but I am worried if I Ieave it as is, the read-only error would creep back. I’m not sure how to troubleshoot this further so I am going to reinstall DietPi.

10

I searched critical target error, dev sdb, sector 5759440 op 0x0: (READ) flags 0x80700 phys_seg 64 prio class 2 ] DMAR: DRHD: handling fault status reg 3

Found

Bad cable? Could also be the grounding (noise) between the 5vdc power supply and the power supply on the external drive…

is smartctl enabled on the drive?

sudo smartctl -i /dev/sdb

might need to start checking the drive…might be on it’s last legs