MotoCache1
Chief Droid Scientist
Two days ago (8/20) over in the SBF to root/unroot topic I was helping a couple people troubleshoot a very strange problem they were having. The result of that troubleshooting led to a very interesting (and important) discovery. Perhaps this has already been discovered by someone else, but if it has, I've not seen it. This discovery may help resolve a lot of bizarre problems that are going on out there.
Symptom:
User successfully flashes a new image to the recovery partition (we'll say it is Clockwork for this example, but it could just as well be SPRecovery or any other recovery). They are able to boot into recovery and their custom recovery is shown as expected. When finished they boot the phone back to the OS. The next time they try to go into recovery, they find that the phone is now back to the stock recovery. Both users received FRG01B OTA and are unrooted. They attempted to install SPRecovery using my new "recovery only" SBF that does not touch the kernel (/boot) partition. This would also occur if they had rooted with EasyRoot and then used ROM Manager to flash Clockwork or SPRecovery on (tested on my phone after resolution was known). It was observed that their phones were reporting a boot loader version of 2C.7C whereas my phone (and the phone of everyone else who could not reproduce the problem) was 2C.6C.
Outcome:
It turns out that the FRG01B OTA package contains a new feature that I have decided to call "Flash Recovery Service" (hereinafter "FRS") after I dug through the OS and found the code that was responsible. FRS knows what the SHA1 hash (similar to an MD5 sum or hash) should be for the recovery partition. When you boot the OS, FRS checks the SHA1 hash of the recovery partition, and if it is not correct it "patches" it back to stock. Hence, you can change the recovery to whatever you want, but at the next boot of the OS it will change back.
How to Fix It:
Many aren't interested in the technical details of how it was isolated or how it works, and will stop after this section.
To keep FRS from flashing your recovery partition back to stock on each boot of the OS, you need to rename or delete the /system/recovery-from-boot.p file so that FRS can't find it. There are many ways to do this. I will present two manual methods here which both require root.
[I have built a method to fix this which does not require root and which will go ahead and root your phone in the process. I will be publishing that information soon and will update this post with a link when it is published.]
Using Root Explorer:
[Remainder of this is not for folks who don't want to understand how any of this works.]
Troubleshooting Highlights:
Problem initially could not be duplicated on my Droid 1 even after flashing FRG01B on using the SBF. Flashing the FRG01B SBF onto my phone did not change the boot loader from 2C.6C to 2C.7C. I was ultimately able to get my phone to boot loader 2C.7C by flashing ESE81 on via SBF and then applying the FRG01B OTA (update.zip). Once I did that, I was able to reproduce the problem they were seeing. Since applying the FRG01B SBF did not exhibit the problem and applying the OTA update.zip did, my first effort at a crude fix was just to flash the SBF back on. This did get rid of the "feature" but did not change the boot loader back to 2C.6C. This told me that the boot loader had nothing to do with the issue. Then I sifted through the OTA update.zip file looking at what the code was doing and found the rest of the pieces of the puzzle.
How Flash Recovery Service Works:
Under FRG01B, in the init.rc file there appears something new that wasn't there in ESE81:
Each time the OS boots it therefore runs /system/etc/install-recovery.sh.
install-recovery.sh:
What this does it look at the first 2K block of the recovery partition image and take a SHA1 has of it. If the hash isn't the one shown then it calls "applypatch". Applypatch uses the first 2,875,392 bytes of the boot partition image (headers, kernel, and ramdisk) and the contents of /system/recovery-from-boot.p (a "diff" file) to mathematically reconstruct an image of the recovery partition. I went through this process myself and indeed ended up with a byte-perfect "stock" recovery image. Then the re-created stock recovery image is flashed to the recovery partition.
By making the diff file (recovery-from-boot.p) unavailable the applypatch can't run and it has no choice but to leave the recovery partition "as is". It will still put an entry in the system log each time you boot that says "Installing new recovery image", but it won't actually be able to do it. If you want, you can also rename or delete the /system/etc/install-recovery.sh file and then it won't even try. In my instructions I don't have you do that just because if you ever want to re-enable it, it's easier if you've only renamed the .p file and just have to rename it back.
Interestingly, even though the version of FRG01B in the SBF does not exhibit the FRS feature, it does contain it. The "service" commands are included in the init.rc in the FRG01B SBF, but interestingly the /system/recovery-from-boot.p and /system/etc/install-recovery.sh files are not. So, the service tries to run but doesn't because the files it needs aren't there.
I have dissected the ramdisk out of the ESE81 kernel (where the init.rc file is located) and ESE81 did not have this "service".
That was a fun one.
[Edited to add: If you flash a custom ROM or kernel that will also kill the FRS. Since the applypatch command uses data from the boot image coupled with the recovery-from-boot.p file, if the boot image doesn't match the SHA1 hash the applypatch will abort as well. This may be why fewer people have noticed -- many of the people who root also put on a custom ROM or kernel shortly thereafter. In some further looking through prior OTA's, the recovery-from-boot.p and install-recovery.sh files were actually in the ESE81 OTA too. The "service" wasn't in the init.rc, but presumably they got kicked off a different way (and it isn't worth going back and figuring out what that way was at the moment).
If you go back far enough there used to be a recovery.img in /system that got flashed in on each reboot. So it seems like this scheme has been around in some form for a long time. I'd bet that with each new OTA there is a round of people having strange problems putting on a custom recovery and having it revert, and they somehow get it fixed and everyone forgets about it. I did some searches and found several such posts here. I read them grinning because everyone is confounded as to how you could flash on a recovery and have it just disappear and go back to stock. Eventually the OP magically fixes it but they don't know how. Usually how they fixed it was by flashing in the recovery within ROM Manager and flashing in a ROM within ROM Manager at the same time. That replaces the boot image with one with a "wrong" SHA1 signature and voila the recovery partition stops reverting. Of course they don't know that was why. Anyway, these discoveries certainly make the presence of this feature in the 2.2 OTA a lot less insidious. It's still important that everyone understand it so they can avoid all the thrashing around trying to solve some odd problem of their recovery not "sticking". I'd still like to know why the boot loader needed to change from 2C.6C to 2C.7C. That probably worries me more than the FRS.]
Symptom:
User successfully flashes a new image to the recovery partition (we'll say it is Clockwork for this example, but it could just as well be SPRecovery or any other recovery). They are able to boot into recovery and their custom recovery is shown as expected. When finished they boot the phone back to the OS. The next time they try to go into recovery, they find that the phone is now back to the stock recovery. Both users received FRG01B OTA and are unrooted. They attempted to install SPRecovery using my new "recovery only" SBF that does not touch the kernel (/boot) partition. This would also occur if they had rooted with EasyRoot and then used ROM Manager to flash Clockwork or SPRecovery on (tested on my phone after resolution was known). It was observed that their phones were reporting a boot loader version of 2C.7C whereas my phone (and the phone of everyone else who could not reproduce the problem) was 2C.6C.
Outcome:
It turns out that the FRG01B OTA package contains a new feature that I have decided to call "Flash Recovery Service" (hereinafter "FRS") after I dug through the OS and found the code that was responsible. FRS knows what the SHA1 hash (similar to an MD5 sum or hash) should be for the recovery partition. When you boot the OS, FRS checks the SHA1 hash of the recovery partition, and if it is not correct it "patches" it back to stock. Hence, you can change the recovery to whatever you want, but at the next boot of the OS it will change back.
How to Fix It:
Many aren't interested in the technical details of how it was isolated or how it works, and will stop after this section.
To keep FRS from flashing your recovery partition back to stock on each boot of the OS, you need to rename or delete the /system/recovery-from-boot.p file so that FRS can't find it. There are many ways to do this. I will present two manual methods here which both require root.
[I have built a method to fix this which does not require root and which will go ahead and root your phone in the process. I will be publishing that information soon and will update this post with a link when it is published.]
Using Root Explorer:
- Navigate to /system
- Remount the partition r/w
- Delete the file or rename it recovery-from-boot.p.not (or whatever you want).
- Remount the partition r/o
- adb shell
- su
- mount -o remount,rw -t yaffs2 /dev/block/mtdblock4 /system
- cd /system
- mv recovery-from-boot.p recovery-from-boot.p.not (or "rm recovery-from-boot.p" if you want to delete it)
- mount -o remount,ro -t yaffs2 /dev/block/mtdblock4 /system
[Remainder of this is not for folks who don't want to understand how any of this works.]
Troubleshooting Highlights:
Problem initially could not be duplicated on my Droid 1 even after flashing FRG01B on using the SBF. Flashing the FRG01B SBF onto my phone did not change the boot loader from 2C.6C to 2C.7C. I was ultimately able to get my phone to boot loader 2C.7C by flashing ESE81 on via SBF and then applying the FRG01B OTA (update.zip). Once I did that, I was able to reproduce the problem they were seeing. Since applying the FRG01B SBF did not exhibit the problem and applying the OTA update.zip did, my first effort at a crude fix was just to flash the SBF back on. This did get rid of the "feature" but did not change the boot loader back to 2C.6C. This told me that the boot loader had nothing to do with the issue. Then I sifted through the OTA update.zip file looking at what the code was doing and found the rest of the pieces of the puzzle.
How Flash Recovery Service Works:
Under FRG01B, in the init.rc file there appears something new that wasn't there in ESE81:
Code:
service flash_recovery /system/etc/install-recovery.sh
oneshotinstall-recovery.sh:
Code:
#!/system/bin/sh
if ! applypatch -c MTD:recovery:2048:e341bb90b74f7eb644b7d72501be659ce229bb1b; then
log -t recovery "Installing new recovery image"
applypatch MTD:boot:2875392:d104d2ec84a2d0660e786c0fb8174bfacb4079d6 MTD:recovery 57de46b6424e717d68c9c3856b339271bc0e5980 3125248 d104d2ec84a2d0660e786c0fb8174bfacb4079d6:/system/recovery-from-boot.p
else
log -t recovery "Recovery image already installed"
fiBy making the diff file (recovery-from-boot.p) unavailable the applypatch can't run and it has no choice but to leave the recovery partition "as is". It will still put an entry in the system log each time you boot that says "Installing new recovery image", but it won't actually be able to do it. If you want, you can also rename or delete the /system/etc/install-recovery.sh file and then it won't even try. In my instructions I don't have you do that just because if you ever want to re-enable it, it's easier if you've only renamed the .p file and just have to rename it back.
Interestingly, even though the version of FRG01B in the SBF does not exhibit the FRS feature, it does contain it. The "service" commands are included in the init.rc in the FRG01B SBF, but interestingly the /system/recovery-from-boot.p and /system/etc/install-recovery.sh files are not. So, the service tries to run but doesn't because the files it needs aren't there.
I have dissected the ramdisk out of the ESE81 kernel (where the init.rc file is located) and ESE81 did not have this "service".
That was a fun one.
[Edited to add: If you flash a custom ROM or kernel that will also kill the FRS. Since the applypatch command uses data from the boot image coupled with the recovery-from-boot.p file, if the boot image doesn't match the SHA1 hash the applypatch will abort as well. This may be why fewer people have noticed -- many of the people who root also put on a custom ROM or kernel shortly thereafter. In some further looking through prior OTA's, the recovery-from-boot.p and install-recovery.sh files were actually in the ESE81 OTA too. The "service" wasn't in the init.rc, but presumably they got kicked off a different way (and it isn't worth going back and figuring out what that way was at the moment).
If you go back far enough there used to be a recovery.img in /system that got flashed in on each reboot. So it seems like this scheme has been around in some form for a long time. I'd bet that with each new OTA there is a round of people having strange problems putting on a custom recovery and having it revert, and they somehow get it fixed and everyone forgets about it. I did some searches and found several such posts here. I read them grinning because everyone is confounded as to how you could flash on a recovery and have it just disappear and go back to stock. Eventually the OP magically fixes it but they don't know how. Usually how they fixed it was by flashing in the recovery within ROM Manager and flashing in a ROM within ROM Manager at the same time. That replaces the boot image with one with a "wrong" SHA1 signature and voila the recovery partition stops reverting. Of course they don't know that was why. Anyway, these discoveries certainly make the presence of this feature in the 2.2 OTA a lot less insidious. It's still important that everyone understand it so they can avoid all the thrashing around trying to solve some odd problem of their recovery not "sticking". I'd still like to know why the boot loader needed to change from 2C.6C to 2C.7C. That probably worries me more than the FRS.]