Building AOSP-A Honest Confession
After weeks of effort, I did it. I successfully compiled a fork of Android 9 OS. The process was so hard but I’m glad that I reached the end. In this story, let’s discuss what went right, what went wrong, and most importantly what to keep in mind when adventuring into this arena.
Should you read this?
As always let us see if the article is worth your time. If you are a developer planning to work or build the AOSP, you should definitely skim through this. Because the topics discussed here can really save you a lot of time (we are talking in weeks). Everyone else, this is optional 😉
Why I wrote this?
I was never a fan of debugging things that involved build-time errors. In fact, if I get any build-time errors, I google most of the time because I feel “Build Time Errors” do not involve any logic and are bugs/incompatibility issues. Although that held mostly true even here, the entire episode involved many other things that a typical android developer (like me) would not expect or want to happen.
Generally, when I hit a dead-end, I take one of the below approaches
1. Ask for help
2. Look for a workaround
3. Quit
However in this case,
1. Ask for help
Is not possible because this is not a work project and none of my tech friends know about this
2. Look for a workaround
Is not possible because every workaround is creating new issues.
3. Quit
Is not possible because, in order to do the actual stuff that I planned, the forked OS has to be flashed on the device !!
So, as you might notice, my only option was to do this no matter how boring it felt. During the process, I felt so frustrated. In fact, that was the real motivation for writing down the article. I know somewhere someone has gone through this and someone will go through this. The idea is to make it less painful by sharing my experience.
Background
I worked on AOSP at “work” two years ago at a startup. At that time I had support from other engineers, the hardware of my choice, and somewhat straightforward documentation for the process.
Recently I got an ODROID N2Plus (SBC) that can run Android. I wanted to hack on it for implementing a few ideas. The first step was to flash the OS with some modifications on the device.
And for flashing it, I needed to build it 😱
The Preparation
From my previous experience, I know compiling the AOSP code is a PITA. Given that it (Android 7 fork) took ~4 hours on average with very good computing power, the first thing I wanted to check was if there are any hardware hacks or smart ways about the build setup. Because if it takes so much time on a Macbook with 64 Gigs of RAM / 1TB of HDD, one of the below should hold true
- Build process itself was horribly designed
- I am missing smart cuts about the build
- All developers working on AOSP have a Fugaku clone.
So, I asked for help on Linkedin.
A Clue
Most of the responses were mostly recommendations. So, I researched more and found an excellent article on XDA Devs. The thing that stood out about the article was that all the recommendations were solidly backed by data. A very interesting read but for TLDR folks, here is the takeaway
“ Disk IO is the biggest bottleneck in building AOSP ”
I always thought about upgrading my PC with a more powerful CPU and memory hoping it to solve the problem but never considered a faster Disk. Because TBH I have no idea about the build process. And when we think about a powerful pc, we tend to think about RAM / Processor always.
Following the article, I invested in a 1 TB SSD instead.
Top 10 Issues
Instead of writing like a timeline or a detailed story, I prefer to keep it lightweight AMAP describing the problems I faced, the resolutions, and the takeaways to have a less terrible experience.
- Do Not clone the repo as a root user
We usually tend to “sudo” stuff on receiving any permission errors. I did the same because the new SSD was mounted as a restricted partition. Remember, if the repo is cloned as a root user, many of the commands in the build process will fail with permission errors. - Getting Stuck with “Chown” on extended hard drives
The mount point needs to follow a certain path for a user to have Write privileges.
More details - Recursive chmod will not work on SYMLINKS
There is a hell of a lot of symlinks inside the source code. So, if you clone the code as a root user (i..e ignore the above suggestions),
Brace Yourselves !!
You’ll see countless meaningless permission errors (actually meaningful after 3 days) during the build process.
More Details - Makefile Tab Issue 🤦♂️
If you edit a make file, make sure the formatting is correct. Because “four spaces don’t make a tab, only a tab makes a tab”
More Details - No such file exists — Even if the file exists
This feels like an impossible error but actually will happen if certain dependencies are not installed. Although I feel the error message can be made less confusing.
More Details - Device Busy — Preventing UnMount
Sometimes there can be processes hidden, especially if you have a lot of parallel stuff going on. They can prevent unmounting the disk.
More Details - Broadcom Wifi Driver Build Failure
Make sure the built environment is exactly the recommended one. If the documentation asks to use a build chain of the version released in the year 2013, use it.
More Details
DO NOT DARE to use anything other than the mentioned versions. - All Warnings are treated as Errors
Track down the line where this happens and remove the appropriate flag that causes this.
More Details - Ubuntu 18.04 Import
I’ll admit, I neither understand what this error means nor what the fix does.
More Details - Race Condition
The worst thing one wants to encounter in an unknown codebase “ is an issue due to race condition”. Because it takes a long time and confidence to declare something happening due to a race condition if we know nothing about it. And yes, we have it here 😭
More Details
How did the Build Process Go
Coming to the interesting part, it took ~5 hours for a clean build every time. Clean build in the sense starting from scratch with a clean build (i..e output) directory. I monitored the RAM and DiskI/O usages.
Here are some captured metrics
Ram Availability (in MB) vs Time
%util (Disk I/O Utilization%) vs Time
Read the definition of the above metrics here
Observe that
1. We always have a free RAM of ~500MB+
2. The utilization of Disk is very high touching 100% sometimes**
I cannot confidently deduce anything from the RAM data because it can very likely be the case where a bit more RAM could have triggered more (parallel) work and thus more RAM utilization.
But the Disk Util% data is interesting. We have used it at a much higher capacity. So, as mentioned in the article before,it looks like DISK I/O is a big (if not the biggest) bottleneck.
**Note: Although this is an SSD, the entire data is present in this DISK only. Hence I don’t expect any of the parallelization issues to come into play here. Even if they have, we see very strong evidence that a lot of DISK I/O is happening.
My knowledge of iostats or Linux for that matter is not good. So, please correct me if I’m wrong
Now, What to keep in mind
1. Have Lots of Patience
This is no joke. Before anything else, you need to mentally prepare for the rough journey ahead.
Unlike Android’s developer documentation, the AOSP build documentation doesn’t explain most details. This is terrible especially for people not familiar with OS or kernel programming. The worst nightmare for a developer is to debug something about which they have neither logical nor semantic knowledge. Imagine staring at a C error log that is 200 lines deep with no useful info
i..e something like this “ target XXX failed blah blah …”
Believe me, you’ll encounter a lot of such things during the journey. The error logs are huge and most times don’t make sense if not you trace back deeply.
So, I repeat “ Have lots of patience”. Don’t expect to pull it off in a couple of days.
2. Have the exact recommended Environment
It is very important to have the EXACT setup as recommended by the documentation. Like toolchains / libraries / dependencies /OS.
No matter how old or obsolete a recommendation might appear, just follow it. I attempted to use the latest versions of the recommended libs / OS and had to spend many days before giving up and switching back to the recommended ones.
Following the recommended set up ensures that we don’t create problems that are not expected to happen in the first place. And remember, a recommendation in a document is made only after it is tested.
If you have to take only one thing from the article, let it be this
3. Always Search for SmartCuts
It is very important to find shortcuts where ever applicable. The build process is really long(~5hrs). So, make sure to learn any related tools or utility functions as listed in “Build Section Here” . For example, If we need to make a minor change, we can just build that module instead of the “Whole source”. Even if we assume that the build system intelligently used the existing build, building the single module is undoubtedly the best time saver approach. Moreover you can skip the entire flashing chore if you know about the awesome “adb sync” command.
More Details in this article
Tips & Tricks
- Observing Build Progress
RN there is no way to view build progress. But once you have built the code, you can follow the below approach to get an idea about the build progress from the second time.
More Details - Which IDE to use for working on AOSP
This is something I am yet to find any concrete answer for. Few people suggest using Android Studio, but personally, I didn’t like it. I find VSCode more suitable for this. Given that I just need an editor and not a very android specific IDE - I don’t know anything about modifying AOSP, where do I start
Congrats, there is no official resource for this. This means you need to come up with innovative ways to figure out stuff. As a good start, try modifying existing code and observe/learn from the changes.
For example, Can you attempt the following
1. Change the welcome message on the boot screen from “ Android is Starting” to “ Hello World .”
2. Change the title of option “Network & Internet” to “Connectivity” in the Settings app.
3. Change the device name from the generic version to something else in the Settings app.
For all the above,
What I did was to search for those existing strings in the codebase and change them. During that process, I got a fair understanding of the code for the settings app and the boot launcher screen. This is my way of getting to know more about the codebase.
Also, you can always refer to the “AOSP code” to check the commits which help a great deal to understand the purpose of a specific change !!
For example, see this commit.
Here is my first change (changed Welcome string to my mother tongue)
Wait, all the work for this change ??
Nope, not at all. There are a lot of things to do with this little monster. My intention is to create a low-cost educational pc from this but that is a long way to go. Right now, I am using this as a media server and for navigating through the 90s mushroom kingdom 🍄. Also for the electronic geeks, this has GPIO pins !!
Next Steps …
While hacking with this, I intend to write down more about AOSP coding given that there are really no good tutorials for the same. Hope to make some progress in this space.
Thanks for reading 😃
Happy New Year and Stay Safe !!
