Hacking erroneous output of SBF Codec

[fish1552]

W.a.g.

Ok, you said jumping right out at us, soooo......

I can see the 2 at the end and it looks like it says "=xx". I tried looking at the characters using the number substitution for them, but it didn't really tell me much.

7C & 7D = 7+12 7+13
A6 & 87 = 10+6 8+7

Other than the last 2 offsets are nearly the same much like the time stamp ones. So it looks like the 4 in the middle are the ones that throw us off so much?

Or am I WAAAYYYY off base and not getting any of it. :icon_eek:

 

[fish1552]

It looks like some of the bytes are inverted

Good guess. Not it. Somewhere we know exactly what to replace some of these bytes with (and the data happens to match the "known good" file).

I see all the FF FF FF just like we had in the BMP file and CG files when we had to delete the portions.

 

[MotoCache1]

Ok, you said jumping right out at us, soooo......

I can see the 2 at the end and it looks like it says "=xx". I tried looking at the characters using the number substitution for them, but it didn't really tell me much.

7C & 7D = 7+12 7+13
A6 & 87 = 10+6 8+7

Other than the last 2 offsets are nearly the same much like the time stamp ones. So it looks like the 4 in the middle are the ones that throw us off so much?

Or am I WAAAYYYY off base and not getting any of it. :icon_eek:

I can see you searching for patterns. This is kind of a side note, but let me tell you about "hex" math.

If you have the number 7C, you're right that the first digit is obviously 7, and the second digit is "worth" 12, but there's something missing there.

A quick delve back into base 10 since we're all familiar with it.

If have have the number 9 it's value is "9". But if I have the number "94", it's not worth 9 + 4 (13). It is actually worth 9x10 + 4*1 because the first column is our "tens" column and the second column is our "ones" column. Most people don't remember (or never knew) but the way we know how much each column is worth is to start at the rightmost column and work to the left.

The right most column is worth 10^0 (read that as 10 with an exponent of zero). The next column is worth 10^1, then 10^2, 10^3 and so on, forever. Well, any number raised to the zero power is equal to 1. It's one of those "identity" laws or something. Then 10^1 = 10, 10^2 = 100, 10^3 = 1000, and so on.

So it is in hexadecimal. The rightmost column is worth 16 ^ 0 (which equals 1), then 16 ^ 1 (16), then 16 ^ 2 (256), and so on.

So, 7C isn't 7+12 (19), it's 7*16 + 12*1 = (112 + 12) = 124.

OK, now that we've been through all that, I'll go ahead and tell you it has nothing to do with what we're solving. But now you understand hex math a little more.

OK, how about a hint and then I'm headed to bed and we can pick up tomorrow. Some of those bytes match something in the OP.

Have a good night!

 

[Jharmon12]

i knew it had to do something with the check sum but i wasnt sure how to calculate the right one because thats the only difference between the good file and the bad one. The question is how much of the difference between the files really matter?

 

[fish1552]

^^^ Bad choice of characters in my substitution of the numbers. I wasn't really adding them, just figuring how they were holding places sort of to search for the pattern. The "+" meaning "&" really. And I guess with my including the "=" it didn't help.

I saw the

File: 0x[SIZE=5][B][COLOR=Red]7C87[/COLOR][/B][/SIZE], Phone: 0x[B][SIZE=5][COLOR=Red]18BB[/COLOR][/SIZE][/B]

Is that actually TELLING us to put the 18 BB where the 7C 87 are.

That is just too easy if it is right. 'Course, you DID hit on the "0x" part and you said something about not emphasizing it....

 

[Jharmon12]

^^^ Bad choice of characters in my substitution of the numbers. I wasn't really adding them, just figuring how they were holding places sort of to search for the pattern. The "+" meaning "&" really. And I guess with my including the "=" it didn't help.

I saw the

File: 0x[SIZE=5][B][COLOR=Red]7C87[/COLOR][/B][/SIZE], Phone: 0x[B][SIZE=5][COLOR=Red]18BB[/COLOR][/SIZE][/B]

Is that actually TELLING us to put the 18 BB where the 7C 87 are.

That is just too easy if it is right. 'Course, you DID hit on the "0x" part and you said something about not emphasizing it....

Ok, i tried what your hypothesis and changed highlighted bytes to bb 18 and saved the sbf and it flashed with no problem (yeah i know playing with fire) But this raises the question, with every sbf that is made do we have to flash it first and then see what the phone expects it to be and then hex edit the proper checksum in like we did with this one, and if thats it what were the other bytes that didnt match up with the good file? Attached is the sbf I changed those two bytes and flashed.

Edit: And if that is the problem then why is the checksum switched around insted of 18BB its BB18?

 

[MotoCache1]

I saw the

File: 0x[SIZE=5][B][COLOR=Red]7C87[/COLOR][/B][/SIZE], Phone: 0x[B][SIZE=5][COLOR=Red]18BB[/COLOR][/SIZE][/B]

Is that actually TELLING us to put the 18 BB where the 7C 87 are.

That is just too easy if it is right. 'Course, you DID hit on the "0x" part and you said something about not emphasizing it....

Ok, i tried what your hypothesis and changed highlighted bytes to bb 18 and saved the sbf and it flashed with no problem (yeah i know playing with fire) But this raises the question, with every sbf that is made do we have to flash it first and then see what the phone expects it to be and then hex edit the proper checksum in like we did with this one, and if thats it what were the other bytes that didnt match up with the good file? Attached is the sbf I changed those two bytes and flashed.

Edit: And if that is the problem then why is the checksum switched around insted of 18BB its BB18?

Obviously I didn't actually go to bed because I'm a moron, but that's beside the point...

So let me ask you a question ... how awesome did it feel when that SBF -- the one that previously would temporarily trash a phone -- flashed right in with no trouble at all? Pretty awesome right? Now who really cares about the nifty picture that it puts on the screen -- that's incidental. What matters is that 6 hours ago (when I first started writing this, more like 7 now I guess) the only tool you had for replacing CG42 was a sledge hammer, and now you have a scalpel -- and you learned something kinda cool too. This was a really great "get your feet wet" hacking exercise because honestly we started out so close to functional that we didn't have to go far, and we also had a working, but otherwise identical specimen. Having someone to guide the way who knew the answer before we ever started didn't hurt either. But now you kind of see the method -- or at least a method -- of figuring this sort of thing out just by brute force observation and reasoning.

So let me tackle your very good questions.

I'm going to take the easy one first:

Edit: And if that is the problem then why is the checksum switched around insted of 18BB its BB18?

The answer to this has to do with Gulliver's Travels -- yes, the book. Rather than explain it I'll just link you to a Wikipedia article about it. The short answer is that sometimes the MSB (most significant byte -- the one that is worth more) is stored first and sometimes the LSB (least significant byte) is stored first. There are various reasons for this, but that doesn't really matter. When thinking of MSB, think of it like this. When we write 5000, how do you know that we mean 5x1000 + 0x100 + 0x10 + 0x1 and not 5x1 + 0*10 + 0*100 + 0*1000? I mean it's a pretty big difference. We know because when we humans deal with decimal numbers we do so in Big Endian fashion (see the Wiki) -- the more important numbers come first. We could write 5000 as 0005 if we all agreed we were going to operate in a Little Endian fashion.

Anyway, when us humans read numbers -- be they hex or decimal or whatever, we always do it Big Endian. So if you see a hex number written 0x7C87 you can be confident that they are talking about 7*16^3 + C*16^2 + 8*16^1 + 7*16^0. But that doesn't mean that's the way the number will be stored in memory or even in a file, depending on if the thing that is going to use it is Big Endian or Little Endian. Of course we don't have any documentation, so let's say somehow we figured out how to calculate the right checksum. How do we know whether to put it in 7C 87 or 87 7C? Well, you know now by observation. You have an error message that says that the file read 0x7C87, but you looked at the file and it actually said 87 7C, so you know that in the future, if you know the right checksum, you need to put it in the file in Little Endian format.

OK, enough of that.

Your first (and probably most important) question was:

But this raises the question, with every sbf that is made do we have to flash it first and then see what the phone expects it to be and then hex edit the proper checksum in like we did with this one, and if thats it what were the other bytes that didnt match up with the good file?

The answer is yes, no, and maybe -- all at once. If you went no further than where we are at now, you would have to do exactly what you just said. Let SBF Codec build your SBF with the bad checksums (it's just keeping the checksum for the prior contents by the way -- I checked), then flash it onto your phone - then get the error log from RSD Lite to find out what the checksum should be, and fix your file. Honestly, this is what lots of people do. It is reliable (obviously RSD Lite knows how to calculate the checksum), and trashing CG42 shouldn't pose any risk of bricking your phone though it is a bit disconcerting to see that code corrupt error. Another option would be to see if you can figure out how to calculate the checksum yourself. This might be a pretty tall order.

We're dealing with a 16 bit checksum. How do I know that? There are lots of ways to come up with the answer. First, the checksum is a 4 digit hex number, right? For me, I know two hex characters are one byte, and we have 4, so that's two bytes, and a byte is 8 bits, so that's 16 bits. Another way to tell would be to bring up the Windows calculator and put it in scientific mode. Then switch it to hex, and then key in the largest possible 4 digit hex number (FFFF). Then switch it to binary. FFFF will change to 1111111111111111. If you count the 1's you'll find 16 of them -- 16 bits.

So, anyway, normally the way you calculate a 16 bit checksum is just to add up the values of every byte in the entire file, and then take a modulus of that total against 0x10000 (0xFFFF + 1). The "modulus" is just the remainder when you divide one number by another. So, 7 modulus 3 is 4. And 25 modulus 2 is 1. Anyway, that's the normal way to calculate a 16 bit checksum and if it were that easy, that would be great. I'll save you the hassle of experimenting -- they aren't calculated that way for the Droid.

The "direct observation" way would be to start with a CG42 that was filled with all zeroes, flash it into the phone, get error, see what the right checksum is, write it down, then change the first zero in CG42 to a 1 and do the whole thing again and see how the checksum changes. Keep incrementing bytes one at a time and watching how the checksum changes. If this were a standard 16 bit checksum the pattern would be that every time you added the value of the next byte the checksum would increase by that much until it got to 0x10000 at which time it would reset to 0x0000 and start counting up again. But it doesn't work that way for the Droid (I've already done all of this if you can't tell).

So, I guess that's the next question for you guys: How would you go about figuring out how to calculate the checksums for the Droid? The direct observation method I just described will work if you are very very good at math. Remember doing differential equations back in calculus and taking big formulas and having to factor them and blah blah blah -- this is kind of the reverse of that. All you are seeing are the answers and you have to figure out what formula is producing them.

Let me know what you come up with. It's possible I don't even have the answer and I just do the RSD Lite trick like everyone else does. Hard to say...

The other question (I was going to ask it, but you asked it first so I pose it back to you -- reworded and fleshed out a little) is:

OK, so out of 10 mismatched bytes, 4 were the datestamp which didn't matter, and 2 were the CG42 checksum (which did matter). That leaves 4 more that were different, but don't seem to be a barrier to flashing. What do they do? Good question. I can tell you that one or more of them causes a difference you will see on the screen in RSD Lite. That's a pretty big hint. Put your hacking caps back on and let me know what you find.

Anyway, good work guys. I hope you feel a little (or big) sense of accomplishment. And with that, now I am going to bed since I need to be up in about 3 hours.

 

[Corinacakes]

Ok :shocked: I'm way too dumb to even look in this thread...........





:rofl3:

 

[MotoCache1]

Ok :shocked: I'm way too dumb to even look in this thread........... :rofl3:

Now now, it's not really that bad is it?

 

[Moto Droid]

I saw the

File: 0x[SIZE=5][B][COLOR=red]7C87[/COLOR][/B][/SIZE], Phone: 0x[B][SIZE=5][COLOR=red]18BB[/COLOR][/SIZE][/B]

Is that actually TELLING us to put the 18 BB where the 7C 87 are.

That is just too easy if it is right. 'Course, you DID hit on the "0x" part and you said something about not emphasizing it....

Ok, i tried what your hypothesis and changed highlighted bytes to bb 18 and saved the sbf and it flashed with no problem (yeah i know playing with fire) But this raises the question, with every sbf that is made do we have to flash it first and then see what the phone expects it to be and then hex edit the proper checksum in like we did with this one, and if thats it what were the other bytes that didnt match up with the good file? Attached is the sbf I changed those two bytes and flashed.

Edit: And if that is the problem then why is the checksum switched around insted of 18BB its BB18?

Obviously I didn't actually go to bed because I'm a moron, but that's beside the point...

So let me ask you a question ... how awesome did it feel when that SBF -- the one that previously would temporarily trash a phone -- flashed right in with no trouble at all? Pretty awesome right? Now who really cares about the nifty picture that it puts on the screen -- that's incidental. What matters is that 6 hours ago (when I first started writing this, more like 7 now I guess) the only tool you had for replacing CG42 was a sledge hammer, and now you have a scalpel -- and you learned something kinda cool too. This was a really great "get your feet wet" hacking exercise because honestly we started out so close to functional that we didn't have to go far, and we also had a working, but otherwise identical specimen. Having someone to guide the way who knew the answer before we ever started didn't hurt either. But now you kind of see the method -- or at least a method -- of figuring this sort of thing out just by brute force observation and reasoning.

So let me tackle your very good questions.

I'm going to take the easy one first:
The answer to this has to do with Gulliver's Travels -- yes, the book. Rather than explain it I'll just link you to a Wikipedia article about it. The short answer is that sometimes the MSB (most significant byte -- the one that is worth more) is stored first and sometimes the LSB (least significant byte) is stored first. There are various reasons for this, but that doesn't really matter. When thinking of MSB, think of it like this. When we write 5000, how do you know that we mean 5x1000 + 0x100 + 0x10 + 0x1 and not 5x1 + 0*10 + 0*100 + 0*1000? I mean it's a pretty big difference. We know because when we humans deal with decimal numbers we do so in Big Endian fashion (see the Wiki) -- the more important numbers come first. We could write 5000 as 0005 if we all agreed we were going to operate in a Little Endian fashion.

Anyway, when us humans read numbers -- be they hex or decimal or whatever, we always do it Big Endian. So if you see a hex number written 0x7C87 you can be confident that they are talking about 7*16^3 + C*16^2 + 8*16^1 + 7*16^0. But that doesn't mean that's the way the number will be stored in memory or even in a file, depending on if the thing that is going to use it is Big Endian or Little Endian. Of course we don't have any documentation, so let's say somehow we figured out how to calculate the right checksum. How do we know whether to put it in 7C 87 or 87 7C? Well, you know now by observation. You have an error message that says that the file read 0x7C87, but you looked at the file and it actually said 87 7C, so you know that in the future, if you know the right checksum, you need to put it in the file in Little Endian format.

OK, enough of that.

Your first (and probably most important) question was:

But this raises the question, with every sbf that is made do we have to flash it first and then see what the phone expects it to be and then hex edit the proper checksum in like we did with this one, and if thats it what were the other bytes that didnt match up with the good file?

The answer is yes, no, and maybe -- all at once. If you went no further than where we are at now, you would have to do exactly what you just said. Let SBF Codec build your SBF with the bad checksums (it's just keeping the checksum for the prior contents by the way -- I checked), then flash it onto your phone - then get the error log from RSD Lite to find out what the checksum should be, and fix your file. Honestly, this is what lots of people do. It is reliable (obviously RSD Lite knows how to calculate the checksum), and trashing CG42 shouldn't pose any risk of bricking your phone though it is a bit disconcerting to see that code corrupt error. Another option would be to see if you can figure out how to calculate the checksum yourself. This might be a pretty tall order.

We're dealing with a 16 bit checksum. How do I know that? There are lots of ways to come up with the answer. First, the checksum is a 4 digit hex number, right? For me, I know two hex characters are one byte, and we have 4, so that's two bytes, and a byte is 8 bits, so that's 16 bits. Another way to tell would be to bring up the Windows calculator and put it in scientific mode. Then switch it to hex, and then key in the largest possible 4 digit hex number (FFFF). Then switch it to binary. FFFF will change to 1111111111111111. If you count the 1's you'll find 16 of them -- 16 bits.

So, anyway, normally the way you calculate a 16 bit checksum is just to add up the values of every byte in the entire file, and then take a modulus of that total against 0x10000 (0xFFFF + 1). The "modulus" is just the remainder when you divide one number by another. So, 7 modulus 3 is 4. And 25 modulus 2 is 1. Anyway, that's the normal way to calculate a 16 bit checksum and if it were that easy, that would be great. I'll save you the hassle of experimenting -- they aren't calculated that way for the Droid.

The "direct observation" way would be to start with a CG42 that was filled with all zeroes, flash it into the phone, get error, see what the right checksum is, write it down, then change the first zero in CG42 to a 1 and do the whole thing again and see how the checksum changes. Keep incrementing bytes one at a time and watching how the checksum changes. If this were a standard 16 bit checksum the pattern would be that every time you added the value of the next byte the checksum would increase by that much until it got to 0x10000 at which time it would reset to 0x0000 and start counting up again. But it doesn't work that way for the Droid (I've already done all of this if you can't tell).

So, I guess that's the next question for you guys: How would you go about figuring out how to calculate the checksums for the Droid? The direct observation method I just described will work if you are very very good at math. Remember doing differential equations back in calculus and taking big formulas and having to factor them and blah blah blah -- this is kind of the reverse of that. All you are seeing are the answers and you have to figure out what formula is producing them.

Let me know what you come up with. It's possible I don't even have the answer and I just do the RSD Lite trick like everyone else does. Hard to say...

The other question (I was going to ask it, but you asked it first so I pose it back to you -- reworded and fleshed out a little) is:

OK, so out of 10 mismatched bytes, 4 were the datestamp which didn't matter, and 2 were the CG42 checksum (which did matter). That leaves 4 more that were different, but don't seem to be a barrier to flashing. What do they do? Good question. I can tell you that one or more of them causes a difference you will see on the screen in RSD Lite. That's a pretty big hint. Put your hacking caps back on and let me know what you find.

Anyway, good work guys. I hope you feel a little (or big) sense of accomplishment. And with that, now I am going to bed since I need to be up in about 3 hours.

When Iam president you can be my top advisor!! Awsome thread:icon_ banana:

 

[fish1552]

Not sure if this matters or not, but trying to play around with FHRED to show line numbers so I can figure out where the problem is with my sbf and hexedit it, I went to OPTIONS > BINARY MODE and it gave me little-endian (Intel) and big-endian (Motorola) as options.

Also, anyone know how to set it up to show the line numbers? I could see the red entries in the files we compared, but I can't figure out where the hell those are in this one by line number.

Also mine has numerous other differences for some reason. Large chunks where the non 0 characters were in your original ones. I modified your DroidForums logo only sbf CG42 file with the image hex from my image, then saved, opened the sbf again and replaced it with my CG42 I saved and got the infamous code corrupt instead of the expected "failed". Well, RSDLite said failed, but phone said corrupt code.

Also, I noticed in the sbf you posted in the other thread (MC1_A855...) it had 2 other files. SBFData.dat & SBFHeader.hdr What are those and if they are needed, how do we get them into SBCodec? It only sees the smg files.

Also, another attempt and I successfully got something I've never gotten before. A "Critical Error: 1337 510B" A battery pull lets me re-flash your DroidForums sbf and everything is like it was on the phone before. This was using the perl script to generate the cg file which I swapped with the CG42 file in your sbf MC1_A855... Now I just want my DROID by Motorola back.... Could I swap the CG42 from my original file with the one in yours and have it work? Or should i just delete everything but the 2 files from mine and figure out the "Line 587" part?

I was sitting here laughing and the wife had to ask me what was wrong and when I told her she gave me one of those "When you screw up your phone..." comments- which I promptly ignored.

Moto, your first signature line really is true. With everything I've done, I would have expected to have bricked this darn thing by now. NO SUCH ummm, luck?

 

[MotoCache1]

Also, anyone know how to set it up to show the line numbers? I could see the red entries in the files we compared, but I can't figure out where the hell those are in this one by line number.

I installed Frhed and mine shows the offsets by default. In fact I don't even see a way to turn that on/off. A hex editor is useless without it. What version do you have? I have 1.7.1.

Also mine has numerous other differences for some reason. Large chunks where the non 0 characters were in your original ones. I modified your DroidForums logo only sbf CG42 file with the image hex from my image, then saved, opened the sbf again and replaced it with my CG42 I saved and got the infamous code corrupt instead of the expected "failed". Well, RSDLite said failed, but phone said corrupt code.

If you're comparing my file to a file you've built yourself with a different image, of course you'll get tons of non-matching bytes -- since the majority of the file is the BMP image and you put a different one in there. Maybe I misunderstood what you're saying.

Also, I noticed in the sbf you posted in the other thread (MC1_A855...) it had 2 other files. SBFData.dat & SBFHeader.hdr What are those and if they are needed, how do we get them into SBCodec? It only sees the smg files.

Those files aren't real. SBF Codec makes them to hold meta data (data about data) that was in the SBF file.

Also, another attempt and I successfully got something I've never gotten before. A "Critical Error: 1337 510B" A battery pull lets me reflash your DroidForums sbf and everything is like it was on the phone before. This was using the perl script to generate the cg file which I swapped with the CG42 file in your sbf MC1_A855... Now I just want my DROID by Motorola back.... Could I swap the CG42 from my original file with the one in yours and have it work? Or should i just delete everything but the 2 files from mine and figure out the "Line 587" part?

If you want your Motorola logo back, just download my "stock logo" SBF and flash it on without making any changes to it. I don't really have any clue about the remainder.

I was sitting here laughing and the wife had to ask me what was wrong and when I told her she gane me one of those "When you screw up your phone..." coments- which I promptly ignored.

Wives. Enough said.

Moto, your first signature line really is true. With everything I've done, I would have expected to have bricked this darn thing by now. NO SUCH ummm, luck?

If you make the right mistake you'll prove my signature line wrong. It can be bricked -- and messing with SBFs is the best way to do it. Here's to hoping none of us ever end up there.

 

[fish1552]

I installed Frhed and mine shows the offsets by default. In fact I don't even see a way to turn that on/off. A hex editor is useless without it. What version do you have? I have 1.7.1.

I can see the offsets, but I was looking more for "001, 002, 003...587" or something. As it is, I figure out how to find line 587 short of counting 587 lines down from the top. Or do I have to figure it out using hex math?

If you're comparing my file to a file you've built yourself with a different image, of course you'll get tons of non-matching bytes -- since the majority of the file is the BMP image and you put a different one in there. Maybe I misunderstood what you're saying.

No, you were right. I just have no idea why the hell I came to that. Apples and oranges.

If you want your Motorola logo back, just download my "stock logo" SBF and flash it on without making any changes to it.

No, sorry. I meant I want to make my sbf I had in the other thread and get that on there using this method. Just trying to work through where I went wrong with my file. I'm positive it's just a matter of finding and changing the parts we talked about in this thread. Most likely it's right in front of my face and I'm just not seeing it. Might just need to step away for a bit and try later.

 

[Jharmon12]

Ok I have tried one on my own and i edited the check sum but had a small problem.

14:49:37,  September 09, 2010
Line: 587
ERROR: Flash failure: Interface AP-OS: Error verifying Code Group 42 checksums. [B][SIZE="4"][COLOR="Red"]File: 0x620F, Phone: 0x62F[/COLOR][/SIZE][/B] (Error Code: 31),
Detailed Error Details: Direction of the Error=No Direction, Command Value=4000000, Code Group Number=42
File: X:\test_dev_usb\flash\code\flashdll\FlashHdlr.cpp
Device ID: 0

As you can see the checksum the phone is expecting is only 3 chars long, Ive tried putting a 0F and a F0 and no go. Im sure its just a hex noob problem on my behalf but alas its a problem.

Edit: Thinking it may have been an error i made a cg42 file both ways, with your script and the tried and true way with the same checksum expectation.

 

[jlutz555]

will SBF-ReCalc fix the checksum errors?