Homindroid
Member
I have posted a few Sapphire 1.1.1 benchmarks on my blog. Check it out and if you have any suggestions please pass them my way!!
http://homindroid.wordpress.com
http://homindroid.wordpress.com
DroidForums.net | Android Forum & News
This is a sample guest message. Register a free account today to become a member! Once signed in, you'll be able to participate on this site by adding your own topics and posts, as well as connect with other members through your own private inbox!
Sapphire Benchmarks
- Thread starter Homindroid
- Start date
ok, since you are going through all of the trouble to test out all of these kernels, you might want to account for all of the variables that can change the results, to make your benchmark tests truly accurate. because on your device without those variables being defined the benchmark scores are nothign more than numbers that no one else can duplicate.
you need things like apps installed, apps opened in background, setcpu settings, ~temp+-X degrees, specific kernel downloaded and installed, voltages used in those kernels, speed settigns for each slot in said kernels, etc.
I like what you are doing, because a true comparison needs to be made between kernels. but a benchmark alone without info for other people to be able to duplicate your results is useless. as it only says hey guys here is what my phone can do. take the scientific approach, define every variable you can possibly think of, make those variables as consistent as possible between each test, and then post your results. because then any joker that comes along is able to duplicate your set variables and compare his phone's performance vs your phone's performance, as each chip is unique it will also give people a good judge of how "perfectly" made their processor is.
a good person to reference should you choose to persue this fully as i have detailed is Skull_One from the overclocking 101 thread, he has both CS and EE degrees. he can name more variables that need to be defined probably.
you need things like apps installed, apps opened in background, setcpu settings, ~temp+-X degrees, specific kernel downloaded and installed, voltages used in those kernels, speed settigns for each slot in said kernels, etc.
I like what you are doing, because a true comparison needs to be made between kernels. but a benchmark alone without info for other people to be able to duplicate your results is useless. as it only says hey guys here is what my phone can do. take the scientific approach, define every variable you can possibly think of, make those variables as consistent as possible between each test, and then post your results. because then any joker that comes along is able to duplicate your set variables and compare his phone's performance vs your phone's performance, as each chip is unique it will also give people a good judge of how "perfectly" made their processor is.
a good person to reference should you choose to persue this fully as i have detailed is Skull_One from the overclocking 101 thread, he has both CS and EE degrees. he can name more variables that need to be defined probably.
Stock kernel in Sapphire 1.1.1 is 250-800 MHz, standard voltage. You have this mislabeled in your post as 1 GHz. I was confused at first as to why the stock kernel would take so much longer to calculate pi (the other kernels all give roughly the same result, as one would expect).
Homindroid
Member
Sniffle,
Thankyou for the excellent reply! I am always game to improve the process and make the benchmarks as analytical as possible!
As far as programs go, I only install what I need to perform screen captures (drocap) and the benchmarks themselves. That said, I WILL now document what apps and processes are running while performing the benchmarks. I also have chosen 1 ghz as the standard and only kernels listed as "low voltage" are maked as such. (IE- ChevyNo1 LV 1ghz). All kernels are curerently downloaded from ROM Manager with the exception of P3's kernal, which is downloaded from his site. I will looking to provide links to the specific kernels I use.
I will try and get in touch with Skull_One as see if he will give me some guidance on how to "tighten" things up in regards to the benchmarks.
Again, thanks for the critique and info!
Dustin
edit to add: I will be updating my procedures blog post to fully reflect how I perform these benchmarks. I think it will make the process easier to tweak if all is known.
Thankyou for the excellent reply! I am always game to improve the process and make the benchmarks as analytical as possible!
As far as programs go, I only install what I need to perform screen captures (drocap) and the benchmarks themselves. That said, I WILL now document what apps and processes are running while performing the benchmarks. I also have chosen 1 ghz as the standard and only kernels listed as "low voltage" are maked as such. (IE- ChevyNo1 LV 1ghz). All kernels are curerently downloaded from ROM Manager with the exception of P3's kernal, which is downloaded from his site. I will looking to provide links to the specific kernels I use.
I will try and get in touch with Skull_One as see if he will give me some guidance on how to "tighten" things up in regards to the benchmarks.
Again, thanks for the critique and info!
Dustin
edit to add: I will be updating my procedures blog post to fully reflect how I perform these benchmarks. I think it will make the process easier to tweak if all is known.
Homindroid
Member
PDroid, thanks! That slipped by me! Post updated.
Dustin
Dustin
Cool - but might I recommend also including cvpcs' 1 GHz kernel in your comparisons.
Cheers.
Homindroid
Member
PDroid, I will add that kernel later today (Hopefully). Off to work
First let me commend you for trying to test a very complicated situation. But alas, the reason I have never done this test is due to the amount of time it takes to make the test "scientifically sound" and "reproduce-able".
Your Linpack results should have less than a .5% difference between kernels of the same speed under the same condition of testing. If we remove Sapphires 800Mhz kernel from your result, you have a 2% difference between your highest and lowest value. Since linpack is purely a CPU derived benchmark, the same CPU tested with a kernel that has the same speed should give the same exact set of results.
Here is what is needed, in my opinion, to get a valid set of benchmarks:
1) Install the ROM clean and DO NOT activate it.
2) Put phone in airplane mode.
3) Delete every application from system/app and data/app except what is needed to boot the phone and get the launcher running.
4) Install all testing apps from a SD card and then remove that SD card.
5) Disable all apps from auto-starting under any condition.
6) Use the performance governor.
7) Each test should be running a minimum of 10 times to make sure your variance in testing is less than .5%. If it is greater, you left something running from either step 3 or 5.
Now if you want to test battery runs, you have to go even further.
1) Fully charge the battery with a wall charger.
2) Delete /data/system/batterystats.bin.
3) Start a battery tracker with 1 minute logging.
4) Set the screen to 100% power and no timeout.
5) Start the test.
6) Unplug the phone from the charger.
7) Wait for the battery to read a minimum of 70%.
8) Stop the test.
9) Pull the log data for the battery and use the data from 90% down to 80%.
One day I will find the time to do this kind of detailed testing. But so far, the state of ROM development has precluded me from doing. Simply put, ROM development is still too much of a moving target at present.
Homindroid
Member
Skull_One,
I will adopt these procedures for all future benchmarks. Noob Question...Does anyone know what apps must be running to boot properly? (Step 3) and how to disable apps from autostarting? Everything else seems straight forward.
Dustin
I will adopt these procedures for all future benchmarks. Noob Question...Does anyone know what apps must be running to boot properly? (Step 3) and how to disable apps from autostarting? Everything else seems straight forward.
Dustin
The last time I did a wipe of all apps was back in the 2.1 days. So I don't have a current list. If I get a chance in the next day or so, I will try to compile a list for the Sapphire ROM.
I use autostarts from the market to keep apps from being able to auto launch based on system events.
I have started making a clean room ROM. Got a baseline install up and running, tonight or tomorrow during my lunch break, I will start removing the remaining apps and lock down the apps that can auto start.
Homindroid,
If you use Clockwork Mod for backup/restores, when I finish getting it to "clean room" status, do you want a copy of it?
Homindroid,
If you use Clockwork Mod for backup/restores, when I finish getting it to "clean room" status, do you want a copy of it?
Homindroid
Member
Skull_One,
Definately! I have already backup'd my the apps I need for benchmarks to my sd card and bought the "autostarts" app. One though, If I do not activate or sign in, how do I restore those apps from the sd card? Do I push a file manager through ADB? Her is the apps I plan on using;
I really can't wait to get started with the new "Clean Room" Sapphire! I really appreciate all the help!
I'm going back to work, teaching till 4. I'll check this thread around 3pm.
Dustin
Definately! I have already backup'd my the apps I need for benchmarks to my sd card and bought the "autostarts" app. One though, If I do not activate or sign in, how do I restore those apps from the sd card? Do I push a file manager through ADB? Her is the apps I plan on using;
- Benchmark PI
- CPU Benchmark
- Linpack
- Quadrant
- autostarts
- astro/root explorer (maybe?)
- Quadrant
I really can't wait to get started with the new "Clean Room" Sapphire! I really appreciate all the help!
I'm going back to work, teaching till 4. I'll check this thread around 3pm.
Dustin
I push Root Explorer right off the bat. Then I install autostarts, SetCPU, Battery Graph, drocap2, OSMonitor and aLogcat. With that baseline installed, I can then kill off the remaining apps that aren't needed for booting, kill the auto launch stuff and then with OSmonitor and aLogcat confirm that the phone is nearly brain dead
