A Linux kernel update is normally routine. Packages are installed, the server reboots, and the new kernel takes over without any visible interruption.
Occasionally, however, the new kernel cannot complete the boot process. The server may stop inside the initial RAM filesystem, enter emergency mode, or display an error similar to:
Unable to mount root filesystem
or:
Failed to mount /sysroot
On systems that use LVM, the affected root device may appear as:
/dev/mapper/ol-root
or:
/dev/mapper/ol00-root
This article explains how to recover an Oracle Linux server after a newer Unbreakable Enterprise Kernel, or UEK, fails to mount the root filesystem. It also shows how to make the last known working kernel the permanent GRUB default without immediately removing the newer kernel.
The procedure is relevant to Oracle Linux 8 and Oracle Linux 9 systems that use GRUB 2, Boot Loader Specification entries, LVM, and the grubby command.
The incident scenario
The server in this anonymized example had more than one UEK version installed.
A newly installed kernel was selected automatically during reboot, but it failed early in the boot process because the root logical volume could not be mounted. The server was recovered by selecting the previous kernel from the GRUB menu.
After recovery, the operating system was confirmed to be running with the older, known-good UEK build. The boot log showed that the working kernel image had successfully started the initramfs and continued into systemd.
The immediate recovery objective was therefore:
- Keep the server running with the known-good kernel.
- Prevent the non-working kernel from being selected automatically.
- Preserve the failed kernel for later investigation.
- Avoid making unrelated storage or application changes until boot stability was restored.
Why a new kernel may fail to mount the root filesystem
The Linux kernel does not initially have access to the complete operating system installed on disk.
During early boot, it loads a temporary filesystem called the initramfs. The initramfs must contain everything required to locate and mount the real root filesystem, including:
- storage-controller drivers;
- NVMe, SATA, RAID or SCSI drivers;
- LVM and device-mapper support;
- filesystem modules such as XFS or ext4;
- encryption support, when LUKS is used;
- the correct root-device parameters;
- scripts used to activate logical volumes.
A server can therefore fail before reaching the normal operating system even when the root filesystem itself is still intact.
Common causes include:
An incomplete or damaged initramfs
The initramfs associated with the new kernel may be missing an important storage, LVM or filesystem module.
A kernel regression
A newer kernel may contain a regression affecting the storage controller, NVMe device, RAID controller, device mapper or another component required during boot.
Incorrect root-device information
The GRUB entry or initramfs may reference an incorrect UUID, logical volume or device path.
Driver compatibility problems
Third-party storage, security, virtualization or hardware drivers may not have been rebuilt correctly for the new kernel.
Interrupted package installation
An incomplete update can install the kernel package without successfully generating all associated boot files.
Separate filesystem damage
A filesystem problem can also prevent mounting. However, when the same root volume boots successfully with an older kernel, a new-kernel or initramfs problem becomes more likely than complete disk failure.
That distinction is important. Booting the older kernel does not prove the storage is perfect, but it demonstrates that the root filesystem is at least readable with the previous kernel and driver stack.
Step 1: Boot the previous kernel from GRUB
When the server fails to boot, restart it and open the GRUB menu.
Depending on the server firmware, console and boot speed, the menu may appear automatically. Otherwise, press Esc repeatedly during startup. Some BIOS-based systems may respond to the Shift key instead.
Look for an entry similar to:
Advanced options for Oracle Linux
Inside that menu, select the previous kernel version.
For example:
Oracle Linux Server, with Unbreakable Enterprise Kernel 6.12.0-104...
instead of:
Oracle Linux Server, with Unbreakable Enterprise Kernel 6.12.0-200...
Do not select rescue mode unless the regular older kernel also fails.
For a remote server, use one of the following console methods:
- physical keyboard and monitor;
- IPMI;
- iLO;
- iDRAC;
- KVM-over-IP;
- virtual machine console;
- cloud serial console.
SSH is not sufficient because the operating system has not yet started.
Step 2: Confirm the active kernel
After the server starts, log in and check the running kernel:
uname -r
Example:
6.12.0-104.43.4.3.el9uek.x86_64
Also check the full system information:
hostnamectl
or:
cat /etc/os-release
uname -a
Do not assume that selecting a GRUB menu entry worked. Always verify the active kernel with uname -r.
Step 3: List all installed kernels
On Oracle Linux, list installed UEK packages with:
rpm -q kernel-uek
To include installation dates:
rpm -q kernel-uek --last
List the kernel images stored in /boot:
ls -lh /boot/vmlinuz-*
List the associated initramfs files:
ls -lh /boot/initramfs-*
A typical server may contain several entries:
/boot/vmlinuz-6.12.0-104.43.4.3.el9uek.x86_64
/boot/vmlinuz-6.12.0-200.x.y.el9uek.x86_64
Keeping at least one known-good older kernel is essential. Removing every older kernel immediately after an update eliminates the easiest recovery path.
Step 4: Check the current GRUB default
Oracle Linux provides the grubby utility for managing boot entries.
Display the currently selected default kernel:
sudo grubby --default-kernel
Example:
/boot/vmlinuz-6.12.0-200.x.y.el9uek.x86_64
If that is the kernel that failed to boot, the default must be changed.
Display all registered kernel entries:
sudo grubby --info=ALL
This shows information such as:
index=0
kernel="/boot/vmlinuz-6.12.0-200.x.y.el9uek.x86_64"
args="ro crashkernel=auto ..."
root="/dev/mapper/ol00-root"
initrd="/boot/initramfs-6.12.0-200.x.y.el9uek.x86_64.img"
title="Oracle Linux Server (...)"
id="..."
Check that the known-good kernel appears in this output before changing anything.
Step 5: Set the known-good kernel as the default
Use the complete /boot/vmlinuz-* path:
sudo grubby --set-default \
/boot/vmlinuz-6.12.0-104.43.4.3.el9uek.x86_64
Replace the version with the actual working kernel installed on the server.
Verify the result:
sudo grubby --default-kernel
Expected output:
/boot/vmlinuz-6.12.0-104.43.4.3.el9uek.x86_64
You can also show the selected boot entry:
sudo grubby --default-index
and:
sudo grubby --info="$(sudo grubby --default-kernel)"
At this stage, the older kernel has been made the persistent default. The newer kernel remains installed but should no longer be selected automatically.
Step 6: Verify the GRUB environment
Check the saved GRUB entry:
sudo grub2-editenv list
Depending on the Oracle Linux configuration, the output may contain a value such as:
saved_entry=<boot-entry-id>
On modern Oracle Linux installations, kernel menu entries may be stored under:
/boot/loader/entries/
List them with:
ls -lh /boot/loader/entries/
To inspect the entries:
grep -R . /boot/loader/entries/
The exact filenames and entry IDs vary by machine. Avoid editing these files manually unless grubby cannot perform the required operation.
Step 7: Reboot and verify the fix
Before rebooting a remote machine, confirm that console access is available. A mistake in the boot configuration could otherwise leave the server inaccessible.
Reboot:
sudo systemctl reboot
After the server returns, verify the active kernel:
uname -r
Also verify that the boot completed normally:
systemctl is-system-running
A healthy system normally returns:
running
Check the current boot log for serious errors:
journalctl -b -p warning..alert --no-pager
Review the kernel command line:
cat /proc/cmdline
Confirm the root filesystem and logical volumes:
findmnt /
lsblk -f
sudo lvs
sudo vgs
sudo pvs
The known-good kernel should now be selected automatically on every reboot.
Do not remove the failed kernel immediately
Once the server is stable, it may be tempting to uninstall the failed kernel immediately.
That is not always the best first step.
Keeping the package temporarily allows you to:
- inspect its boot entry;
- compare initramfs contents;
- regenerate its initramfs;
- retry it after correcting the problem;
- provide package versions in a support request;
- confirm whether a later update supersedes it.
First collect basic information:
rpm -q kernel-uek
rpm -q kernel-uek-core
rpm -q kernel-uek-modules
Check which package owns the failed kernel image:
rpm -qf /boot/vmlinuz-<failed-kernel-version>
Only remove it after the machine has completed at least one successful unattended reboot with the chosen default kernel and after any necessary diagnostic data has been preserved.
How to investigate the failed kernel
Changing the GRUB default restores availability, but it does not identify the original cause.
The following checks help narrow it down.
Compare initramfs sizes
ls -lh /boot/initramfs-*.img
A dramatically smaller initramfs may indicate incomplete generation, although size alone is not proof.
Inspect initramfs contents
Use lsinitrd:
lsinitrd /boot/initramfs-<failed-kernel-version>.img | less
Search for LVM and device-mapper components:
lsinitrd /boot/initramfs-<failed-kernel-version>.img |
grep -Ei 'lvm|device-mapper|dm-|xfs|ext4|nvme'
Compare that with the working kernel:
lsinitrd /boot/initramfs-<working-kernel-version>.img |
grep -Ei 'lvm|device-mapper|dm-|xfs|ext4|nvme'
Validate the boot files
Check that both the kernel and initramfs exist:
ls -lh \
/boot/vmlinuz-<failed-kernel-version> \
/boot/initramfs-<failed-kernel-version>.img
Review package installation history
sudo dnf history
Then inspect the relevant transaction:
sudo dnf history info <transaction-id>
Look for errors, interruptions or failed post-install scripts.
Check available space in /boot
df -h /boot
df -i /boot
A nearly full /boot partition can prevent the initramfs or boot entry from being generated correctly.
Review logs from the kernel installation
journalctl --since "24 hours ago" |
grep -Ei 'kernel|initramfs|dracut|grub|grubby'
Adjust the time range to match the update.
Rebuilding the failed kernel’s initramfs
When the kernel package is complete but the initramfs is suspected, it can be rebuilt with dracut.
First back up the existing image:
sudo cp -a \
/boot/initramfs-<failed-kernel-version>.img \
/boot/initramfs-<failed-kernel-version>.img.backup
Then regenerate it:
sudo dracut \
--force \
/boot/initramfs-<failed-kernel-version>.img \
<failed-kernel-version>
For example:
sudo dracut \
--force \
/boot/initramfs-6.12.0-200.x.y.el9uek.x86_64.img \
6.12.0-200.x.y.el9uek.x86_64
Confirm that the command completed without errors:
echo $?
A result of 0 indicates successful command completion.
Inspect the regenerated file:
ls -lh /boot/initramfs-<failed-kernel-version>.img
Do not immediately make the rebuilt kernel the permanent default. Test it during a controlled maintenance window by selecting it manually from GRUB.
This ensures the server still falls back to the known-good kernel after an unsuccessful test.
Testing the newer kernel safely
A safe test plan should include:
- Confirmed physical or remote console access.
- A documented working kernel.
- The working kernel configured as the default.
- A current backup or snapshot.
- A maintenance window.
- No unrelated storage or application changes during the same reboot.
- A rollback procedure prepared in advance.
Boot the test kernel manually from GRUB rather than changing the permanent default.
After it starts, confirm:
uname -r
findmnt /
systemctl is-system-running
journalctl -b -p warning..alert --no-pager
Only after a successful controlled boot should you consider making the newer kernel the default.
Recovery when no installed kernel boots
When both the new and old kernels fail, use rescue media or the Oracle Linux installation ISO.
The general recovery process is:
- Boot into rescue mode.
- Discover the root logical volume.
- Mount the installed system.
- Mount
/bootand/boot/efiwhen separate. - Bind
/dev,/proc,/sysand/run. - Enter the installed system with
chroot. - rebuild the initramfs or change the GRUB default.
Example discovery commands:
lsblk -f
sudo pvs
sudo vgs
sudo lvs
Activate LVM when necessary:
sudo vgchange -ay
A root volume might then be mounted with a command similar to:
sudo mount /dev/mapper/ol00-root /mnt/sysroot
This path is only an example. Use the actual logical-volume name reported by lvs and lsblk.
When /boot is separate:
sudo mount /dev/<boot-partition> /mnt/sysroot/boot
For UEFI systems:
sudo mount /dev/<efi-partition> /mnt/sysroot/boot/efi
Bind the runtime filesystems:
for path in dev proc sys run; do
sudo mount --bind /$path /mnt/sysroot/$path
done
Enter the installed environment:
sudo chroot /mnt/sysroot
From the chroot, inspect kernels and use grubby or dracut as described earlier.
Because partition layouts differ, rescue-mode operations should be performed carefully and preferably after preserving a backup.
Important mistakes to avoid
Do not delete the working kernel
Always retain at least one verified kernel.
Do not edit GRUB configuration blindly
Use grubby where possible. Manual editing of generated GRUB configuration can be overwritten or may target the wrong file.
Do not run multiple repairs at once
Avoid updating firmware, changing storage configuration, rebuilding initramfs and replacing drivers during the same test. Change one variable at a time.
Do not assume the newest kernel is active
Always check:
uname -r
Do not reboot a remote server without console access
A second boot failure can turn a recoverable problem into a prolonged outage.
Do not assume “unable to mount root” means the disk is dead
The same message can result from missing initramfs drivers, LVM activation failure, an incorrect root argument or a kernel regression.
Recommended kernel-update procedure for production servers
A safer update process is:
Before the update
uname -r
sudo grubby --default-kernel
rpm -q kernel-uek --last
df -h /boot
Confirm that a previous working kernel remains installed.
After installing the update, before reboot
rpm -q kernel-uek --last
sudo grubby --info=ALL
ls -lh /boot/vmlinuz-* /boot/initramfs-*
Confirm that the new kernel and initramfs both exist.
During the first reboot
Use a maintenance window and keep console access available.
After reboot
uname -r
systemctl is-system-running
journalctl -b -p warning..alert --no-pager
For Kubernetes or container hosts, also verify:
kubectl get nodes
kubectl get pods -A
For systems with GPUs or third-party drivers, verify that the appropriate kernel modules loaded successfully.
Frequently asked questions
How do I change the default kernel in Oracle Linux?
Use:
sudo grubby --set-default /boot/vmlinuz-<kernel-version>
Then verify:
sudo grubby --default-kernel
Is it safe to boot an older kernel?
Yes, as a temporary recovery measure, provided it is a trusted kernel already installed on the system and compatible with the current user-space packages.
Security fixes introduced only in newer kernels will not be active while the older kernel is running, so the underlying failure should still be investigated.
Should I remove the kernel that failed?
Not immediately. First stabilize the server, collect diagnostic information, and determine whether the problem is a damaged initramfs, incomplete installation or genuine regression.
What does “unable to mount root filesystem” mean?
It means the early boot environment could not locate, activate, understand or mount the filesystem containing the operating system.
The cause can be storage-related, but it can also be a missing driver, LVM issue, incorrect boot parameter or incomplete initramfs.
Why does the older kernel work with the same disk?
Each kernel normally has its own initramfs, modules and drivers. The older kernel may contain the correct storage support while the new kernel’s boot environment does not.
Does setting an older default prevent future kernel updates?
No. Package updates can still install newer kernels. Administrators should verify the GRUB default after future update transactions, especially until the original issue is resolved.
Conclusion
When an Oracle Linux kernel update prevents the server from mounting its root filesystem, the fastest safe recovery is usually to boot the previous working kernel and make it the temporary GRUB default.
The essential commands are:
uname -r
rpm -q kernel-uek --last
sudo grubby --info=ALL
sudo grubby --set-default /boot/vmlinuz-<working-kernel-version>
sudo grubby --default-kernel
This restores predictable boot behavior without destroying diagnostic evidence or removing the newer kernel prematurely.
The most important operational lessons are straightforward:
- retain a known-good kernel;
- maintain console access;
- verify the active kernel after every reboot;
- check
/bootbefore updates; - use
grubbyrather than editing generated files blindly; - test repaired kernels manually before making them the default;
- change one component at a time.
A failed kernel update does not necessarily mean the operating system or root disk is lost. In many cases, the server can be recovered cleanly by selecting the previous kernel, pinning it as the default, and investigating the newer kernel’s initramfs and driver stack in a controlled maintenance window.
