Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update 6.1 - 6.2 makes RPi4B unbootable #1490

Closed
2 of 7 tasks
done7k opened this issue Aug 3, 2021 · 22 comments
Closed
2 of 7 tasks

Update 6.1 - 6.2 makes RPi4B unbootable #1490

done7k opened this issue Aug 3, 2021 · 22 comments
Labels
board/raspberrypi Raspberry Pi Boards usb-ssd USB SSD related issue

Comments

@done7k
Copy link

done7k commented Aug 3, 2021

Hardware Environment

  • Raspberry Pi [4b], SSD M.2 via USB3
  • ODROID [C2/C4/N2(+)/XU4]
  • ASUS Tinker [S]
  • Generic x86-64 (like Intel NUC)
  • OVA (Open Virtualization Appliance, on Intel NUC or any other hardware, please add the Hypervisor you are using)

Home Assistant OS release:

  • Fresh installation of release x.y
  • Updated from version 6.1
  • Additional information (if accessible):

Supervisor logs:

Journal logs:

Kernel logs:

Description of problem:

  • after upgrade the system does not boot. the green LED blinks a few times only. No messages through HDMI screen.
@agners agners added the board/raspberrypi Raspberry Pi Boards label Aug 3, 2021
@agners
Copy link
Member

agners commented Aug 3, 2021

Which version have you been using before? What type of USB 3.0 M.2 adapter are you using?

6.2 comes with a new firmware, but if 6.1 (with the old firmware) works and 6.2 (with the new firmware) reproducible doesn't work, then the best corse of action is to report the problem upstream with the Raspberry Pi firmware developers.

@done7k
Copy link
Author

done7k commented Aug 3, 2021

yes, it was on 6.1 before. then backup'ed and switched from SD card to USB3 SSD disk (Inteso M.2 SSD TOP SATA III 128Gb, in Gembird - EE2280-U3C-01- enclosure w/o external power supply).
Also, I was experimenting today with images:
Tried a fresh OS 6.2 (32bit) installation - same result - no boot/black screen
tried a fresh OS 6.1 (32bit) installation - fine! then update to 6.2 from console - no boot /black screen;
tried a fresh OS 6.2 (64bit) installation - it started fine but did not restore all add-ons… I replaced to 6.1 32bit

@GiveMeTheBits
Copy link

I also experienced this. I am still restoring because it would not apply my snapshots correctly. That is a separate issue, but I'm bummed out.

Raspberry Pi [4b]
Updated from version 6.1
installed on a Samsung_Portable_SSD_T5
everything else was up to date as of yesterday except OS.

@inverse
Copy link
Contributor

inverse commented Aug 4, 2021

I previously had issues going from 5.12 -> 5.13 in a similar setup but going to 6.1 was stable. Will hold off the upgrade to 6.2 but following this.

@agners
Copy link
Member

agners commented Aug 4, 2021

@GiveMeTheBits are you using 32-bit as well?

@agners agners added the usb-ssd USB SSD related issue label Aug 4, 2021
@mstoffel-sag
Copy link

Hi, I‘m not able to update from 6.1 to 6.2. might that be related? I’m also using an ssd:

21-08-05 06:30:21 ERROR (MainThread) [supervisor.hassos] Home Assistant Operating System update failed with: Installation error: Failed updating slot rootfs.1: Opening output device /dev/disk/by-partlabel/hassos-system1 failed: Device or resource busy

@agners
Copy link
Member

agners commented Aug 5, 2021

@mstoffel-sag Do you have a SD card or something with an old HAOS installation still in that device? Does that still occures after rebooting?

@GauthierCrp
Copy link

GauthierCrp commented Aug 5, 2021

Hello,

Same problem for me, after updating from 6.1 to 6.2, my raspberry pi 4 is unable to boot.
I'm using a SSD. Then I restored from 6.1 on USB stick and try a new update... same result. 6.2 seems broken on RPI4 with USB boot.
Should try with SD card.

I'm waiting for bug fix to update now.

Edit : don't know what this means (sadly not my domain...), but when I plug my usb stick to flash a new image, Windows finds many partitions on the device. Like 2 boot partitions of 24mo, 2 partitions of 256mo, one of 8mo, one of 96mo and then the leaving space with 27go. Too much to be normal isn't it ?

@GiveMeTheBits
Copy link

@GiveMeTheBits are you using 32-bit as well?

I was on 64-bit.

@mstoffel-sag
Copy link

@GauthierCrp I have no sd card (mount only shows sda mountpoints) but its weird. The update worked for me now. Looks like /dev/root at 100% is normal. What I changed though was: I removed a change that I did for mounting external media using udev. Got that from here:
https://gist.github.com/eklex/c5fac345de5be9d9bc420510617c86b5

After I deleted the rule and did a reboot the update worked flawlessly.

Kind Regards,

Marco

@agners
Copy link
Member

agners commented Aug 6, 2021

@mstoffel-sag thanks for the update. I think your issue is unrelated to what others are seeing here, as they have troubles booting after updating. 100% /dev/root is normal, we use a read-only squashfs, which basically is a minimal image for the whole root file system.

That said, the fact that your update succeeded now with USB SSD and others not shows that only some USB SSD installation are affected. Another round of detective work 😢

@GauthierCrp 8 partitions is normal. We use that to separate kernel from the root file system, and to have two installations (A/B system, basically a fallback if booting one fails).

The booting problems are most likely caused by the upgrade to the lastest Raspberry Pi firmware. Unfortunately, its not the first time that previously working USB SSD installation got broken by newer Raspberry Pi firmwares.

To rule out any other problems, can you try the following: Flash 6.1, copy start.elf and start.dat from the first FAT partition (you might have to go to Disk Management or something to assign a letter/mount the partition). Connect the USB SSD back to the Rpi and update to 6.2. Verify boot is broken still. Connect the USB SSD to your PC again and replace start.elf and start.dat with the version from the 6.1 installation. Connect back to the Raspberry Pi and see if that makes 6.2 boot.

@agners
Copy link
Member

agners commented Aug 6, 2021

As always: Not reproducible here 😞

Using Raspberry Pi 4 Model B on a Samsung T1 series (MU-P8250B) attached via USB 3.0. Installed 64-bit HAOS 6.1, upgraded to 6.2 and rebooted successfully.

Bootloader: d6d82cf9 (Feb 16 2021) (disconnect all boot media, this should display the Raspberry Pi bootloader screen)

my_photo-74

@mstoffel-sag
Copy link

Thanks Stefan,

but since this is my "production" system I'm happy that it works.... hope you'll understand that..

@agners
Copy link
Member

agners commented Aug 6, 2021

@mstoffel-sag sure. The firmware test was more meant for others having the issue.

@done7k
Copy link
Author

done7k commented Aug 6, 2021

@agners , Have not you tried 32bits image? or you believe it should not matter? As I mentioned with 64 bits I got 6.2 booted once but after I restored my Hassio configuration it did not restored all my add-ons. And when I started restoring them one by one it did not allowed me saying the system is in FREEZE state.. Thus I decided to move back on the 32bits. 6.1... not sure if it is correct to migrate from 6.1 32bit to 6.2 64bit ..

@agners
Copy link
Member

agners commented Aug 6, 2021

I did not test 32-bit, I'll do that next week.

it did not restored all my add-ons.

Afaik, that should work. Which Add-On did not work? 64-bit is recommended now, so moving to 64-bit would be a good thing 😄

@done7k
Copy link
Author

done7k commented Aug 6, 2021

@agners just have checked my board's bootloader. I flashed a new SD with fresh Raspberian; booted; it did something and rebooted automatically; then I removed SD and rebooted again. My bootloader dated by April 29
20210806_211533

@done7k
Copy link
Author

done7k commented Aug 6, 2021

Afaik, that should work. Which Add-On did not work? 64-bit is recommended now, so moving to 64-bit would be a good thing smile

Adguard_home; AppDaemon_4; Grocy; Zigbee2mqtt .. and some other ones but not very important. For instance I tried to restore AppDaemon and Adguard and got 'System in freeze state' msg.

@GiveMeTheBits
Copy link

@agners , Have not you tried 32bits image? or you believe it should not matter? As I mentioned with 64 bits I got 6.2 booted once but after I restored my Hassio configuration it did not restored all my add-ons. And when I started restoring them one by one it did not allowed me saying the system is in FREEZE state.. Thus I decided to move back on the 32bits. 6.1... not sure if it is correct to migrate from 6.1 32bit to 6.2 64bit ..

This is my experience as well. I assumed it was my Snapshots were incomplete or corrupt, but maybe it is a problem in the update? I was running 6.1 64-bit, updated to 6.2 and it died. wiped my SSD, installed fresh from RPI imager. I unfortunately did not check which OS version it installed, but I think it would pull the current one from github??? At this point, my snapshot restores were missing add-ons and I went back to my most recent full snapshot from March and tried to reapply the partial from July 30th, but I also got freeze state errors and issues with it actually applying anything. I wound up extracting the tarball and just copying my config folder back into my running config. :(

@DonnyDepp
Copy link

Same here... Since the latest update homeassistant became totally unresponsive and is always getting network timeouts.
The network port led is blinking and going out in a loop.

@agners
Copy link
Member

agners commented Sep 3, 2021

Finally came around testing with 32-Bit. I can reproduce the issue. Enabling UART via enable_uart=1 in config.txt shows that U-Boot hangs:

U-Boot 2021.04 (Jul 28 2021 - 22:21:10 +0000)

DRAM:  7.9 GiB
RPI 4 Model B (0xd03114)
MMC:   mmcnr@7e300000: 1, emmc2@7e340000: 0
In:    serial
Out:   serial
Err:   serial
Net:   eth0: ethernet@7d580000
PCIe BRCM: link up, 5.0 Gbps x1 (SSC)
starting USB...
Bus xhci_pci: Host not halted after 16000 microseconds.
probe failed, error -16
No working controllers found
Card did not respond to voltage select! : -110
Card did not respond to voltage select! : -110
starting USB...
Bus xhci_pci: Host not halted after 16000 microseconds.
probe failed, error -16
No working controllers found
USB is stopped. Please issue 'usb start' first.
starting USB...
Bus xhci_pci: Host not halted after 16000 microseconds.
probe failed, error -16
No working controllers found

It seems that enabling compressed kernel support (via #1418) changed U-Boot slightly, which changed timing and causes this issue. Its weird however that unrelated configuration options break boot... I have seen the issue already in the past (see #1203).

agners added a commit to agners/operating-system that referenced this issue Sep 3, 2021
…stant#1490)

Add a patch to fix a U-Boot issue causing troubles when enumerating USB
devices (required when booting from USB devices).
@agners
Copy link
Member

agners commented Sep 3, 2021

It seems that this patch fixes the problem in my case: https://www.mail-archive.com/u-boot@lists.denx.de/msg409182.html

I'll add the patch to HAOS, so this should be resolved with the upcoming 6.3 release.

@agners agners closed this as completed Sep 3, 2021
agners added a commit that referenced this issue Sep 3, 2021
Add a patch to fix a U-Boot issue causing troubles when enumerating USB
devices (required when booting from USB devices).
agners added a commit that referenced this issue Sep 3, 2021
Add a patch to fix a U-Boot issue causing troubles when enumerating USB
devices (required when booting from USB devices).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
board/raspberrypi Raspberry Pi Boards usb-ssd USB SSD related issue
Projects
None yet
Development

No branches or pull requests

7 participants