The Occasional Pain of Upgrading Firmware
This describes some of the problems I encountered when upgrading from FortiOS 5.2.7 to 5.4. I have installed several point upgrades and patch releases to the FortiGate in the past few months, and frankly was a little to trusting about this. I had planned to install 5.4 on Friday evening, then go to dinner with my brother. I didn't anticipate any major problems, but I got them anyway. Fortunately this was with my in-house FortiGate 60-D so I wasn't faced with the prospect of having a client down while I sweated this out. I went to dinner anyway and resumed troubleshooting the next day, but because time was limited it was Sunday before I really had things rocking again.
What I Did Wrong
- I did not read the docs before I installed this upgrade.
- I didn't follow best practices.
- I didn't take my own advice that I give my clients.
I paid for my stupidity. On the plus side, I started this after business hours and I did make a backup of my configuration, not that it helped much. I did try restoring the backup early in the process and it left me in exactly the same place.
After the fact, I went back and closely read the Release Notes for 5.4 and none of the problems I endured were described. You can download the Release Notes on the FortiNet customer service portal where you download the firmware image.
Surprise #1: Version 5.4 of FortiOS apparently completely revamps the Load Balancing algorithm and the mechanisms behind it. These settings DID NOT upgrade cleanly on my system. I hope it works for you, but if you're reading this I have a hunch you might have had problems too. I have two internet providers for my business, for redundancy and to help improve network response. Setting this up was a bit of a pain in version 5.2 but it worked fine when I was finished. The conclusions I'm presenting here are based on my experience with upgrading to the 5.4 firmware.
Before you upgrade to Version 5.4, you must remove all of the prior WAN load balancing configurations. That includes any and all routing and IPv4 policies where the load balancing is configured. If your experience is the same as mine this will not upgrade cleanly, and your internet connection will drop until it is fixed.
So, now that your internet connection is trashed, what do you do? (I'm assuming you're using the GUI admin tool here.)
First remove the routing rules for all WAN load balanced ports, which probably means all of the routing rules. Write them down if you wish, but rip them out.
Second, remove the IPv4 policies that refer to the old WAN load-balancing scheme, which is now trashed anyway.
Third, in the Network interface configuration screen, remove all traces of the definition for the old load balancing scheme, and both of the WAN network interface definitions. Write down any static IP addresses, account logins and passwords, which you should have anyway.
To summarize the above steps, you're about to start fresh.
Now recreate both of the WAN network links. They should both show green arrows, and be up and running. If either of them is showing a red arrow fix it now before continuing.
Note: If you want an interim test connection, when you have the first WAN interface up you may want to temporarily define a routing rule and an IPv4 policy to allow yourself internet access again. Just remember to remove that policy and the routing before continuing to redefine the WAN LLB configuration.
When you try to define the WAN LLB you should have both WAN interfaces available to add to the virtual connection. If either of them is not available to add the the configuration that means some artifact of the previous configuration is still out there somewhere. Find it and remove it.
Now go to the FortiNet Knowledge Base, or click this link to get a document named: "Technical Note: WAN load balance (volume based) and redundant Internet connections". This is KB document #FD38759. I'm not going to repeat the instructions because this document does a very good job of walking you through setting up the new WAN LLB, with the exception of:
Surprise #2: In the above document step #3 has the following line:
"Select Load Balance Algorithm > Volume > set Weight for WAN1 and WAN2."
Did you grasp the importance of that? Neither did I. You absolutely MUST assign percentages to both interfaces where indicated. The program will allow you to leave the percentages at zero, and if both are weighted at zero you will not have any internet access. This is something I learned after several hours of trial, error and frustration.
A lesser gotcha is that you'll need to refer back to the WAN setup for each interface and make a note of the gateway addresses. Then on the WAN LLB definition screen, fill in the gateway address for each link. I'm not certain how critical it is to enter the gateways, but it can't hurt. The important thing to note is the WAN LLB setup program does not fill in that information from the individual WAN setup screens.
Apart from that problem, everything seems to work in the instructions.
As an added thought, I'm not advocating that you should use this particular load balancing technique, just saying that it will get you operational quickly if you're careful. For an excellent discussion comparing different load balancing methods, I suggest you login to the support portal and download the What's New document, which I've also linked, where there is an excellent discussion of this subject. I foresee hours of delightful reading and experimentation ahead of me.
Note: I will also point out that it's obvious 5.4 is red-hot right now. The above knowledge base article was written on 5/30/16, five days before my attempt at upgrading. The What's New PDF was updated today while I was writing this blog article because version 5.4.1 was released. Fortinet is working hard to keep up with the support issues, so check their site early and often, and take full advantage.