Sapphire Benchmarks

[Homindroid]

I have posted a few Sapphire 1.1.1 benchmarks on my blog. Check it out and if you have any suggestions please pass them my way!!

http://homindroid.wordpress.com

[sniffle]

ok, since you are going through all of the trouble to test out all of these kernels, you might want to account for all of the variables that can change the results, to make your benchmark tests truly accurate. because on your device without those variables being defined the benchmark scores are nothign more than numbers that no one else can duplicate.

you need things like apps installed, apps opened in background, setcpu settings, ~temp+-X degrees, specific kernel downloaded and installed, voltages used in those kernels, speed settigns for each slot in said kernels, etc.

I like what you are doing, because a true comparison needs to be made between kernels. but a benchmark alone without info for other people to be able to duplicate your results is useless. as it only says hey guys here is what my phone can do. take the scientific approach, define every variable you can possibly think of, make those variables as consistent as possible between each test, and then post your results. because then any joker that comes along is able to duplicate your set variables and compare his phone's performance vs your phone's performance, as each chip is unique it will also give people a good judge of how "perfectly" made their processor is.

a good person to reference should you choose to persue this fully as i have detailed is Skull_One from the overclocking 101 thread, he has both CS and EE degrees. he can name more variables that need to be defined probably.

[pdroid]

Stock kernel in Sapphire 1.1.1 is 250-800 MHz, standard voltage. You have this mislabeled in your post as 1 GHz. I was confused at first as to why the stock kernel would take so much longer to calculate pi (the other kernels all give roughly the same result, as one would expect).

[sniffle]

pdroid you should put in your signature that you are an EE, so that on things related to kernels people know that you know what you are talking about... :-P

[Homindroid]

Sniffle,

Thankyou for the excellent reply! I am always game to improve the process and make the benchmarks as analytical as possible!

As far as programs go, I only install what I need to perform screen captures (drocap) and the benchmarks themselves. That said, I WILL now document what apps and processes are running while performing the benchmarks. I also have chosen 1 ghz as the standard and only kernels listed as "low voltage" are maked as such. (IE- ChevyNo1 LV 1ghz). All kernels are curerently downloaded from ROM Manager with the exception of P3's kernal, which is downloaded from his site. I will looking to provide links to the specific kernels I use.

I will try and get in touch with Skull_One as see if he will give me some guidance on how to "tighten" things up in regards to the benchmarks.

Again, thanks for the critique and info!

Dustin

edit to add: I will be updating my procedures blog post to fully reflect how I perform these benchmarks. I think it will make the process easier to tweak if all is known.

[Homindroid]

PDroid, thanks! That slipped by me! Post updated.

Dustin

[sniffle]

no problem by doing this you will actually give people a true baseline to compare against, instead of numbers beign thrown around... it's needed to be done for a while now.

[pdroid]

PDroid, thanks! That slipped by me! Post updated.

Dustin

Cool - but might I recommend also including cvpcs' 1 GHz kernel in your comparisons.

Cheers.

[Homindroid]

PDroid, I will add that kernel later today (Hopefully). Off to work

[Skull One]

I have posted a few Sapphire 1.1.1 benchmarks on my blog. Check it out and if you have any suggestions please pass them my way!!

http://homindroid.wordpress.com

First let me commend you for trying to test a very complicated situation. But alas, the reason I have never done this test is due to the amount of time it takes to make the test "scientifically sound" and "reproduce-able".

Your Linpack results should have less than a .5% difference between kernels of the same speed under the same condition of testing. If we remove Sapphires 800Mhz kernel from your result, you have a 2% difference between your highest and lowest value. Since linpack is purely a CPU derived benchmark, the same CPU tested with a kernel that has the same speed should give the same exact set of results.

Here is what is needed, in my opinion, to get a valid set of benchmarks:

1) Install the ROM clean and DO NOT activate it.
2) Put phone in airplane mode.
3) Delete every application from system/app and data/app except what is needed to boot the phone and get the launcher running.
4) Install all testing apps from a SD card and then remove that SD card.
5) Disable all apps from auto-starting under any condition.
6) Use the performance governor.
7) Each test should be running a minimum of 10 times to make sure your variance in testing is less than .5%. If it is greater, you left something running from either step 3 or 5.

Now if you want to test battery runs, you have to go even further.
1) Fully charge the battery with a wall charger.
2) Delete /data/system/batterystats.bin.
3) Start a battery tracker with 1 minute logging.
4) Set the screen to 100% power and no timeout.
5) Start the test.
6) Unplug the phone from the charger.
7) Wait for the battery to read a minimum of 70%.
8) Stop the test.
9) Pull the log data for the battery and use the data from 90% down to 80%.

One day I will find the time to do this kind of detailed testing. But so far, the state of ROM development has precluded me from doing. Simply put, ROM development is still too much of a moving target at present.