Monday, March 28, 2016

Citrix NetScaler VPX appliance fails to boot with the error: “can’t load ‘kernel’”

Problem

You’ve recently upgraded a NetScaler VPX appliance to version 11.0.63.16.nc and have been running the upgrade for a few months but noticed that the appliance fails to boot into the kernel after a restart halting at the following:

FreeBSD/x86 bootstrap loader, Revision NS1.2

Loading /boot/defaults/loader.conf

Unable to load a kernel!

\

can’t load ‘kernel’

Type ‘?’ for a list of commands, ‘help’ for more detailed help.

OK

image

You proceed with a restore of the virtual machine from a backup solution such as Veeam but noticed that all of the backups boot into the same error.

You attempt to try and find the /boot/defaults/loader.conf directory in the disks but it does not appear to exist:

image

lsdev -v

image

set currdev=disk1s1b

image

ls

image

set currdev=disk1s1d

ls

image

set currdev=disk1s1e

ls

image

image

You are able to locate the /nsconfig directory on one of the disks:

image

… and able to list the contents with the command ls nsconfig:

imageimage

… but missing the other directories such as:

  1. /var
  2. /netscaler

*Reference: http://support.citrix.com/article/CTX200418

Solution

One of the ways to resolve this issue is to either restore from a backed up virtual machine that would actually boot properly or redeploy a new appliance then restore configuration files.  If you do not have either then the following steps can be used in an attempt to bring the appliance back to a bootable state:

Execute lsdev -v to list the available drives:

image

From the drives listed, we need to find the one that contains the nsinstall directory and in this example, the list is disk1s1e so execute the following to set the context of the prompt to that drive:

set currdev=disk1s1e

image

Use the ls command to list the directories:

imageimage

Note the nsinstall directory in the screenshot above. Execute the ls nsinstall/ command to list the contents:

image

Note that the following 2 directories are listed:

  1. 10.5.54.9.nc
  2. 11.0.63.16.nc

This appliance was deployed as version 10.5.54.9.nc initially then upgraded several times to version 11.0.63.16.nc and what appears to have happened is that the boot configuration has gone missing.  Execute the commands:

ls 10.5.54.9.nc

imageimage

ls 11.0.63.16.nc

imageimage

image

Note that both directories contain quite a few files. The file we’re interested in is the NetScaler kernel file that is named ns-{version}-{build}.gz. Since both directories contain this file, we’ll attempt to load the newer 11.0.63.16.nc kernel file by executing the following:

unload

load disk1s1e:/nsinstall/11.0.63.16.nc/ns-11.0-63.16

image

With the kernel successfully loaded, proceed to type boot to boot into the kernel:

image

With the appliance booted and accessible, if you don’t have a configuration backup, proceed to get it now through the CLI or GUI:

image

What you’ll notice at this point is that if we attempt to restart the appliance, we’ll encounter the same error message:

can’t load ‘kernel’

… we faced earlier and this is because the boot configuration has been lost.  The next step to ensure that this boot configuration stays persistent is to reinstall the NetScaler binaries by executing the following:

shell

cd /var/nsinstall/

ls

image

cd 11.0.63.16.nc

./installns

image

You should now see the installation begin:

image

Proceed to reboot the appliance once the install has completed:

image

The appliance should now be able to boot into the OS and operate as it should:

image

Note that there is a good chance that:

  1. The licenses on the appliance will need to be regenerated so log onto the web portal and see if the NetScaler is now labeled as NetScaler VPX (1)
  2. SSL Certificates may be missing so they would need to get re-created (the files will still be there)
  3. Various NetScaler features may be disabled so they will need to get re-enabled

2 comments:

Anonymous said...

Great article. Saved doing a complete rebuild today.
Thanks.

AKSHAY PODDAR said...

Great Article. Save my Day.