Compiling custom Coreboot firmware for a Lenovo ThinkCentre M900 to support Intel 7th gen (Kaby Lake)
This was something I worked on a few months ago. It was one of the most challenging projects I've attempted. Lots and lots of trial and error to get a working end result.
Continue at your own peril! These are the chronicles of a girl who did not know how deep the rabbit hole went. She nearly lost everything (ok, not everything but nearly her entire media collection).
You may have been there too. You've picked up your second-hand enterprise machine and eventually find a moment to pop in the new CPU you ordered on eBay. In my case, an i5-7500 with the fabled Intel HD Graphics 630 to support 10bit 4K transcoding on Jellyfin. As you and your friend wait expectantly for the computer to post, your hearts slowly sink and your anxious faces are met with nothing but a black screen.
What went wrong?
Oh dang. According to Google's AI overview this model of PC was firmware locked to prevent CPU upgrades. Thanks capitalism and profit-maxxing CEOs.
So, your custom firmware journey begins. You order an "ACEIRMC SOIC8 SOP8 Flash Chip IC Test Clips Socket Adpter Programmer BIOS + CH341A 24 25 Series for EEPROM Flash BIOS USB Programmer Module (Double Clip+ USB)".
Ok. Cool.
There's a software tool called CoffeeTime which lets you modify an existing stock firmware and inject CPU microcodes to give the motherboard permission to boot from your CPU. Unfortunately, even after several days of tinkering with this community effort, nothing works. There's a thing called Intel Management Engine that at face value is used for enterprise deployment or something. It has to run on the board or it will refuse to work, and apparently it always has COMPLETE access to all memory and hardware. That is nuts. Some people suspect it is an intentional backdoor at a near hardware level. And of course, CVEs have been discovered over time.
You find an EXTREMELY random looking AI-generated looking download site with an M910Q firmware. The flash takes a few minutes, as usual. And, after spinning up the fans a few times, giving a black screen for a few minutes. LOGO!
Wow. Right before taking a break you try booting it again and literally nothing at all happens. You tear yourself away despite the sinking feeling because you have been working on this for hours and hours.
Get some food and come back. It seems that it works after reflashing it, so that is good. Your server runs for several weeks until you are away for a while and your server goes down. You know what has likely happened, despite being a few hundred km away.
OK, so this firmware you found online is not a long term solution. Somehow you realize that you own a M900 and not a M700 like you've been thinking for the past few months... How did you get that wrong. I guess they look pretty similar. Well, this might open the opportunity to use Coreboot because someone has created a configuration for that system.
Coreboot is an open source motherboard firmware implementation which should be CPU agnostic. There are so many configuration options in the CLI though... it feels so unlikely that you are going to be able to get it working.
After 10+ hours of reading forum posts and tinkering, there are several critical pieces that you feel you've figured out. The Coreboot website is really great too.
- ifdtool is used to extract binary blobs from the original firmware that are needed. this includes the flash descriptor, intel management engine, and gigabit ethernet controller. you then configure coreboot to reinsert these into your newly compiled firmware.
- you can try to nerf the management engine blob using a setting in
make menuconfig, but it could potentially also prevent the firmware from functioning. in my case it had to be used intact. - the maximum size for the firmware is 16mb. there are some tools included in the configurator that can shift the regions you assign to the different blobs that allow you to decrease the firmware size if necessary. some of the payload options are larger than others. seaBIOS is the default and small.
The challenge for me was that seaBIOS doesn't support UEFI. I didn't want to reinstall Debian using legacy boot so I really wanted to get UEFI working. edk2 is the alternative, however it is much larger. It was necessary to change the starting address of the payload to get it to fit, as I recall.
The real lesson of this endeavor is to better document the steps and variations I try. I would like to be able to share accurate and helpful information on here so this work can have a shared benefit.
My other lesson was not to believe the Google AI suggestion that it might be a corruption of your BTRFS disk that is preventing Debian from booting. Your fstab file defined disks have to match the hardware exactly or Debian will be unhappy and not boot. The real problem was something to do with my mergefs flags, not any of the underlying BTRFS partitions. I figured this out when I booted into a Debian live environment and the BTRFS partitions appeared just fine.
These were the mergefs flags that ended up working.
defaults,rw,nonempty,allow_other,use_ino,cache.files=off,moveonenospc=true,dropcacheonclose=true,minfreespace=15G,fsname=mergerfs-pool,category.create=mfs,category.action=epall,category.search=ff 0 0I had tried using fsck to repair the BTRFS drive and of course fsck doesn't support BTRFS so it corrupted the drive. I tried repairing it with btrfs check and thank glob it worked. However, I had lingering issues when running backups on restic because the fsck operation had messed up the drive. I eventually reformatted it. It is not fun to spend time thinking you have lost your media library and will need to rebuild it. I had dismissed this particular backup knowing I could rebuild it if totally necessary, but now I realize that the real loss would have been my time. It is now backed up, albeit less regularly than my other drives.
I can share the finished Coreboot binary in case it is helpful for anyone.
Hope this was a journey for you... I will work to better document my future projects.