The follow post shows output obtained and the upgrade process performed recently on a clients QFabric system. This output was captured updating from 12.2X30 to 12.2X50 Junos release via a ‘Non Stop Services Upgrade’ (NSSU) method. This method basically is a very conservative approach updating redundant components one at a time.
The overall process is:
- Upgrade Director Group
- Upgrade QFabric Interconnects
- Upgrade each node group
- Network Node group (NW-NG-01)
- Each redundant server node group (RSNG)
- Each server node group (my client did not have any SNGs)
Before Upgrade Backup
All that is required to be backed up is the QFabric configuration file, everything else about the install is the QFabric standard and able to be restored using documented Juniper methods.
To backup the config log into the device and:
- Capture the output from ‘show configuration | no-more’
or
- ‘show configuration | save QFabric.conf’
- Remotely: scp username@x.x.x.x:/pbdata/packages/QFabric.conf
Upgrade Process with Output
Director Group Upgrade
Copy the RPM image to the director to /pbdata/packages. This process takes around 2 hours. We started at 7:15am and finished at 9:15am.
- scp FILE.rpm root@x.x.x.x:/pbdata/packages
- Log into the DG via the VIP and start the upgrade
- request system software nonstop-upgrade director-group FILE.rpm
- Junos looks in /pbdata/packages by default
Upgrade Output:
| root@FSASYDBRDQFAB01> request system software nonstop-upgrade director-group jinstall-qfabric-12.2X50-D20.4.rpmValidating update package jinstall-qfabric-12.2X50-D20.4.rpmInstalling update package jinstall-qfabric-12.2X50-D20.4.rpmInstalling fabric images version 12.2X50-D20.4Performing cleanupPackage install completeInstalling update package jinstall-qfabric-12.2X50-D20.4.rpm on peer
Triggering Initial Stage of Fabric Manager Upgrade Updating CCIF default image to 12.2X50-D20.4 Updating FM-0 to Junos version 12.2X50-D20.4 [Status 2012-09-24 14:43:37]: Fabric Manager: Upgrade Initial Stage started [FM-0 2012-09-24 14:43:52]: Transferring FM-0 Mastership to LOCAL DG [FM-0 2012-09-24 14:45:44]: Finished FM-0 Mastership switch [NW-NG-0 2012-09-24 14:45:59]: Transferring NW-NG-0 Mastership to LOCAL DG [NW-NG-0 2012-09-24 14:47:22]: Finished NW-NG-0 Mastership switch [FM-0 2012-09-24 14:48:10]: Retrieving package [FM-0 2012-09-24 14:49:13]: Retrieving package [FM-0 2012-09-24 14:50:15]: Pushing bundle to re0 [Status 2012-09-24 14:52:03]: Load completed with 0 errors [Status 2012-09-24 14:52:03]: Reboot is required to complete upgrade [Status 2012-09-24 14:52:04]: Trying to Connect to Node: FM-0 [Status 2012-09-24 14:52:19]: Rebooting FM-0 [FM-0 2012-09-24 14:52:19]: Waiting for FM-0 to terminate Starting Peer upgrade Initiating rolling upgrade of Director peer: version 12.2X50-D20.4 Inform CCIF regarding rolling upgrade [Peer Update Status]: Validating install package jinstall-qfabric-12.2X50-D20.4.rpm [Peer Update Status]: jinstall-qfabric-12.2X50.D20.4-4 [Peer Update Status]: Cleaning up node for rolling phase one upgrade [Peer Update Status]: Director group upgrade complete [Peer Update Status]: COMPLETED [Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade [Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade [Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade [Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade [Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade [Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade [Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade [Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade [Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade [Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade [Peer Update Status]: Waiting for peer to return after reboot and start phase one of rolling upgrade [Peer Update Status]: Waiting for peer to return after reboot and start phase one of rolling upgrade [Peer Update Status]: Waiting for peer to return after reboot and start phase one of rolling upgrade [Peer Update Status]: Waiting for peer to return after reboot and start phase one of rolling upgrade [Peer Update Status]: Waiting for peer to return after reboot and start phase one of rolling upgrade [Peer Update Status]: Waiting for peer to return after reboot and start phase one of rolling upgrade [Peer Update Status]: Waiting for peer to return after reboot and start phase one of rolling upgrade [Peer Update Status]: Waiting for peer to return after reboot and start phase one of rolling upgrade [Peer Update Status]: Waiting for peer to complete phase one of rolling upgrade [Peer Update Status]: Waiting for peer to complete phase one of rolling upgrade [Peer Update Status]: Peer completed phase one of rolling upgrade Setting peer DG node as the master SFC Delaying start of local upgrade to allow peer services time to initialize [15 minutes] Delaying start of local upgrade to allow peer services time to initialize [15 minutes] Delaying start of local upgrade to allow peer services time to initialize [12 minutes] Delaying start of local upgrade to allow peer services time to initialize [9 minutes] Delaying start of local upgrade to allow peer services time to initialize [6 minutes] Delaying start of local upgrade to allow peer services time to initialize [3 minutes] [Peer Update Status]: Check for VMs on dg0 Triggering Final Stage of Fabric Manager Upgrade: Updating FM-0 to Junos version 12.2X50-D20.4 [Status 2012-09-24 15:33:31]: Fabric Manager: Upgrade Final Stage started [NW-NG-0 2012-09-24 15:33:45]: Transferring NW-NG-0 Mastership to REMOTE DG [NW-NG-0 2012-09-24 15:35:08]: Finished NW-NG-0 Mastership switch [Status 2012-09-24 15:35:08]: Upgrading FM-0 VM on worker DG to 12.2X50-D20.4 [DRE-0 2012-09-24 15:36:09]: Retrieving package [DRE-0 2012-09-24 15:37:02]: ------- re0: ------- [Status 2012-09-24 15:38:28]: Load completed with 0 errors [Status 2012-09-24 15:38:28]: Reboot is required to complete upgrade [DRE-0 2012-09-24 15:38:34]: Waiting for DRE-0 to terminate [DRE-0 2012-09-24 15:38:46]: Waiting for DRE-0 to come back [DRE-0 2012-09-24 15:42:00]: Running Uptime Test for DRE-0 [DRE-0 2012-09-24 15:42:06]: Uptime Test for DRE-0 Passed [Status 2012-09-24 15:42:06]: DRE-0 Booted successfully Performing post install shutdown and cleanup Broadcast message from root (Mon Sep 24 15:42:07 2012): The system is going down for reboot NOW! Director group upgrade complete |
Interconnect Upgrade
This process takes around an hour. It will upgrade Junos on each System Control Board (SCB) partition grabbing the code automatically via the FTP running on the active Director Group member. We observed roughly that time starting at 7:15am and finished at 9:15am.
- From the DG CLI initiate the config:
- request system software nonstop-upgrade fabric FILE.rpm
Output:
| [FC-0 2012-09-24 16:22:17]: Retrieving package[FC-1 2012-09-24 16:22:18]: Retrieving package[IC-F7811 2012-09-24 16:22:39]: Retrieving package[IC-F7712 2012-09-24 16:22:41]: Retrieving package[FC-0 2012-09-24 16:23:14]: Validating on re0[FC-1 2012-09-24 16:23:18]: Validating on re0[IC-F7712 2012-09-24 16:23:57]: Pushing bundle to re1
[IC-F7811 2012-09-24 16:23:58]: Pushing bundle to re1 [IC-F7712 2012-09-24 16:24:47]: Validating on re1 [IC-F7811 2012-09-24 16:24:48]: Validating on re1 [FC-0 2012-09-24 16:25:02]: Done with validate on all chassis [FC-0 2012-09-24 16:25:02]: ------- re0: ------- [FC-1 2012-09-24 16:25:11]: Done with validate on all chassis [FC-1 2012-09-24 16:25:11]: ------- re0: ------- [IC-F7712 2012-09-24 16:29:51]: Validating on re0 [IC-F7811 2012-09-24 16:30:48]: Validating on re0 [IC-F7712 2012-09-24 16:34:10]: Done with validate on all chassis [IC-F7712 2012-09-24 16:34:10]: ------- re1: ------- [IC-F7811 2012-09-24 16:34:20]: Done with validate on all chassis [IC-F7811 2012-09-24 16:34:20]: ------- re1: ------- [IC-F7712 2012-09-24 16:34:55]: Step 1 of 20 Creating temporary file system [IC-F7712 2012-09-24 16:34:55]: Step 2 of 20 Determining installation source [IC-F7712 2012-09-24 16:34:55]: Step 3 of 20 Processing format options [IC-F7712 2012-09-24 16:34:55]: Step 4 of 20 Determining installation slice [IC-F7712 2012-09-24 16:34:56]: Step 5 of 20 Creating and labeling new slices [IC-F7811 2012-09-24 16:34:56]: Step 1 of 20 Creating temporary file system [IC-F7712 2012-09-24 16:34:56]: Step 6 of 20 Create and mount new file system [IC-F7811 2012-09-24 16:34:57]: Step 2 of 20 Determining installation source [IC-F7811 2012-09-24 16:34:57]: Step 3 of 20 Processing format options [IC-F7811 2012-09-24 16:34:57]: Step 4 of 20 Determining installation slice [IC-F7811 2012-09-24 16:34:58]: Step 5 of 20 Creating and labeling new slices [IC-F7811 2012-09-24 16:34:58]: Step 6 of 20 Create and mount new file system [IC-F7712 2012-09-24 16:35:04]: Step 7 of 20 Getting OS bundles [IC-F7712 2012-09-24 16:35:04]: Step 8 of 20 Updating recovery media [IC-F7811 2012-09-24 16:35:07]: Step 7 of 20 Getting OS bundles [IC-F7811 2012-09-24 16:35:07]: Step 8 of 20 Updating recovery media [IC-F7712 2012-09-24 16:35:27]: Step 9 of 20 Extracting incoming image [IC-F7811 2012-09-24 16:35:30]: Step 9 of 20 Extracting incoming image [IC-F7712 2012-09-24 16:36:38]: Step 10 of 20 Unpacking OS packages [IC-F7712 2012-09-24 16:36:41]: Step 11 of 20 Mounting jbase package [IC-F7811 2012-09-24 16:36:42]: Step 10 of 20 Unpacking OS packages [IC-F7811 2012-09-24 16:36:45]: Step 11 of 20 Mounting jbase package [IC-F7712 2012-09-24 16:37:05]: Step 12 of 20 Creating base OS symbolic links [IC-F7811 2012-09-24 16:37:09]: Step 12 of 20 Creating base OS symbolic links [IC-F7712 2012-09-24 16:38:03]: Step 13 of 20 Creating fstab [IC-F7712 2012-09-24 16:38:03]: Step 14 of 20 Creating new system files [IC-F7712 2012-09-24 16:38:04]: Step 15 of 20 Adding jbundle package [IC-F7811 2012-09-24 16:38:07]: Step 13 of 20 Creating fstab [IC-F7811 2012-09-24 16:38:07]: Step 14 of 20 Creating new system files [IC-F7811 2012-09-24 16:38:07]: Step 15 of 20 Adding jbundle package [IC-F7712 2012-09-24 16:40:35]: Step 16 of 20 Backing up system data [IC-F7811 2012-09-24 16:40:36]: Step 16 of 20 Backing up system data [IC-F7712 2012-09-24 16:40:37]: Step 17 of 20 Setting up shared partition data [IC-F7811 2012-09-24 16:40:37]: Step 17 of 20 Setting up shared partition data [IC-F7712 2012-09-24 16:40:37]: Step 18 of 20 Checking package sanity in installation [IC-F7712 2012-09-24 16:40:37]: Step 19 of 20 Unmounting and cleaning up temporary file systems [IC-F7811 2012-09-24 16:40:37]: Step 18 of 20 Checking package sanity in installation [IC-F7811 2012-09-24 16:40:37]: Step 19 of 20 Unmounting and cleaning up temporary file systems [IC-F7712 2012-09-24 16:40:40]: Step 20 of 20 Setting da0s1 as new active partition [IC-F7811 2012-09-24 16:40:41]: Step 20 of 20 Setting da0s1 as new active partition [IC-F7712 2012-09-24 16:40:50]: ------- re0: ------- [IC-F7811 2012-09-24 16:40:52]: ------- re0: ------- [IC-F7712 2012-09-24 16:41:36]: Step 1 of 20 Creating temporary file system [IC-F7712 2012-09-24 16:41:36]: Step 2 of 20 Determining installation source [IC-F7712 2012-09-24 16:41:37]: Step 3 of 20 Processing format options [IC-F7712 2012-09-24 16:41:37]: Step 4 of 20 Determining installation slice [IC-F7712 2012-09-24 16:41:38]: Step 5 of 20 Creating and labeling new slices [IC-F7712 2012-09-24 16:41:38]: Step 6 of 20 Create and mount new file system [IC-F7811 2012-09-24 16:41:39]: Step 1 of 20 Creating temporary file system [IC-F7811 2012-09-24 16:41:39]: Step 2 of 20 Determining installation source [IC-F7811 2012-09-24 16:41:40]: Step 3 of 20 Processing format options [IC-F7811 2012-09-24 16:41:40]: Step 4 of 20 Determining installation slice [IC-F7811 2012-09-24 16:41:41]: Step 5 of 20 Creating and labeling new slices [IC-F7811 2012-09-24 16:41:42]: Step 6 of 20 Create and mount new file system [IC-F7712 2012-09-24 16:41:49]: Step 7 of 20 Getting OS bundles [IC-F7712 2012-09-24 16:41:50]: Step 8 of 20 Updating recovery media [IC-F7811 2012-09-24 16:41:51]: Step 7 of 20 Getting OS bundles [IC-F7811 2012-09-24 16:41:51]: Step 8 of 20 Updating recovery media [IC-F7712 2012-09-24 16:42:15]: Step 9 of 20 Extracting incoming image [IC-F7811 2012-09-24 16:42:19]: Step 9 of 20 Extracting incoming image [IC-F7712 2012-09-24 16:44:01]: Step 10 of 20 Unpacking OS packages [IC-F7712 2012-09-24 16:44:04]: Step 11 of 20 Mounting jbase package [IC-F7811 2012-09-24 16:44:05]: Step 10 of 20 Unpacking OS packages [IC-F7811 2012-09-24 16:44:07]: Step 11 of 20 Mounting jbase package [IC-F7712 2012-09-24 16:44:36]: Step 12 of 20 Creating base OS symbolic links [IC-F7811 2012-09-24 16:44:40]: Step 12 of 20 Creating base OS symbolic links [IC-F7712 2012-09-24 16:46:01]: Step 13 of 20 Creating fstab [IC-F7712 2012-09-24 16:46:01]: Step 14 of 20 Creating new system files [IC-F7712 2012-09-24 16:46:01]: Step 15 of 20 Adding jbundle package [IC-F7811 2012-09-24 16:46:06]: Step 13 of 20 Creating fstab [IC-F7811 2012-09-24 16:46:06]: Step 14 of 20 Creating new system files [IC-F7811 2012-09-24 16:46:06]: Step 15 of 20 Adding jbundle package [IC-F7712 2012-09-24 16:49:41]: Step 16 of 20 Backing up system data [IC-F7811 2012-09-24 16:49:45]: Step 16 of 20 Backing up system data [IC-F7811 2012-09-24 16:49:47]: Step 17 of 20 Setting up shared partition data [IC-F7811 2012-09-24 16:49:48]: Step 18 of 20 Checking package sanity in installation [IC-F7811 2012-09-24 16:49:48]: Step 19 of 20 Unmounting and cleaning up temporary file systems [IC-F7811 2012-09-24 16:49:51]: Step 20 of 20 Setting da0s1 as new active partition [IC-F7712 2012-09-24 16:51:13]: Step 17 of 20 Setting up shared partition data [IC-F7712 2012-09-24 16:51:14]: Step 18 of 20 Checking package sanity in installation [IC-F7712 2012-09-24 16:51:14]: Step 19 of 20 Unmounting and cleaning up temporary file systems [IC-F7712 2012-09-24 16:51:17]: Step 20 of 20 Setting da0s1 as new active partition [Status 2012-09-24 16:51:32]: Load completed with 0 errors [Status 2012-09-24 16:51:32]: Reboot is required to complete upgrade [Status 2012-09-24 16:51:32]: Rebooting FC-1 [FC-1 2012-09-24 16:51:33]: Waiting for FC-1 to terminate [FC-1 2012-09-24 16:52:18]: Waiting for FC-1 to come back [FC-1 2012-09-24 16:55:10]: Running Uptime Test for FC-1 [FC-1 2012-09-24 16:55:26]: Uptime Test for FC-1 Passed [Status 2012-09-24 16:55:27]: FC-1 Booted successfully [Status 2012-09-24 16:55:27]: Rebooting FC-0 [FC-0 2012-09-24 16:55:27]: Waiting for FC-0 to terminate [FC-0 2012-09-24 16:56:12]: Waiting for FC-0 to come back [FC-0 2012-09-24 16:59:06]: Running Uptime Test for FC-0 [FC-0 2012-09-24 16:59:22]: Uptime Test for FC-0 Passed [Status 2012-09-24 16:59:22]: FC-0 Booted successfully [Status 2012-09-24 16:59:22]: Rebooting IC-F7811 [IC-F7811 2012-09-24 16:59:28]: Waiting for IC-F7811 to terminate [IC-F7811 2012-09-24 16:59:59]: Waiting for IC-F7811 to come back [IC-F7811 2012-09-24 17:06:45]: Running Uptime Test for IC-F7811 [IC-F7811 2012-09-24 17:07:34]: Waiting for FM to be ready [IC-F7811 2012-09-24 17:13:09]: Performing post-boot Health-Check [IC-F7811 2012-09-24 17:14:24]: Waiting for routes to sync [IC-F7811 2012-09-24 17:14:32]: Uptime Test for IC-F7811 Passed [Status 2012-09-24 17:14:32]: IC-F7811 Booted successfully [Status 2012-09-24 17:14:32]: Rebooting IC-F7712 [IC-F7712 2012-09-24 17:14:34]: Waiting for IC-F7712 to terminate [IC-F7712 2012-09-24 17:15:07]: Waiting for IC-F7712 to come back [IC-F7712 2012-09-24 17:22:03]: Running Uptime Test for IC-F7712 [IC-F7712 2012-09-24 17:22:47]: Waiting for FM to be ready [IC-F7712 2012-09-24 17:29:28]: Performing post-boot Health-Check [IC-F7712 2012-09-24 17:30:43]: Waiting for routes to sync [IC-F7712 2012-09-24 17:30:49]: Uptime Test for IC-F7712 Passed [Status 2012-09-24 17:30:50]: IC-F7712 Booted successfully Success |
Node Group Upgrades
The NWNG took around an hour (for 4 nodes) and around 40 minutes for a RSNG. This process upgrades a node at a time in the group and updates both slices. Currently there is no command to verify each slice’s version, it is a known issue.
Node Groups tested were 1 Network node group and 2 RSNGs:
- NW-NG-0
- RSNG01
- RSNG02
- From the DG CLI initiate the config:
- request system software nonstop-upgrade node-group GROUP-NAME FILE.rpm
Output:
| root@FSASYDBRDQFAB01> ...0-D20.4.rpm node-group NW-NG-0Upgrading target(s): NW-NG-0[NW-NG-0 2012-09-24 17:33:25]: Starting with package ftp://169.254.0.3/pub/images/12.2X50-D20.4/jinstall-qfx.tgz[NW-NG-0 2012-09-24 17:33:25]: Retrieving package[NW-NG-0 2012-09-24 17:34:47]: Pushing bundle to P6172-C[NW-NG-0 2012-09-24 17:35:20]: Pushing bundle to P6136-C[NW-NG-0 2012-09-24 17:35:53]: Pushing bundle to fpc4
[NW-NG-0 2012-09-24 17:36:27]: Pushing bundle to fpc5 [NW-NG-0 2012-09-24 17:36:59]: P6172-C: Validate package... [NW-NG-0 2012-09-24 17:43:31]: P6136-C: Validate package... [NW-NG-0 2012-09-24 17:43:31]: fpc4: Validate package... [NW-NG-0 2012-09-24 17:43:41]: fpc5: Validate package... [NW-NG-0 2012-09-24 17:43:41]: ------- P6172-C ------- [NW-NG-0 2012-09-24 17:44:17]: Step 1 of 20 Creating temporary file system [NW-NG-0 2012-09-24 17:44:17]: Step 2 of 20 Determining installation source [NW-NG-0 2012-09-24 17:44:18]: Step 3 of 20 Processing format options [NW-NG-0 2012-09-24 17:44:18]: Step 4 of 20 Determining installation slice [NW-NG-0 2012-09-24 17:44:18]: Step 5 of 20 Creating and labeling new slices [NW-NG-0 2012-09-24 17:44:19]: Step 6 of 20 Create and mount new file system [NW-NG-0 2012-09-24 17:44:27]: Step 7 of 20 Getting OS bundles [NW-NG-0 2012-09-24 17:44:27]: Step 8 of 20 Updating recovery media [NW-NG-0 2012-09-24 17:44:48]: Step 9 of 20 Extracting incoming image [NW-NG-0 2012-09-24 17:46:02]: Step 10 of 20 Unpacking OS packages [NW-NG-0 2012-09-24 17:46:07]: Step 11 of 20 Mounting jbase package [NW-NG-0 2012-09-24 17:46:33]: Step 12 of 20 Creating base OS symbolic links [NW-NG-0 2012-09-24 17:47:33]: Step 13 of 20 Creating fstab [NW-NG-0 2012-09-24 17:47:33]: Step 14 of 20 Creating new system files [NW-NG-0 2012-09-24 17:47:34]: Step 15 of 20 Adding jbundle package [NW-NG-0 2012-09-24 17:50:07]: Step 16 of 20 Backing up system data [NW-NG-0 2012-09-24 17:50:08]: Step 17 of 20 Setting up shared partition data [NW-NG-0 2012-09-24 17:50:09]: Step 18 of 20 Checking package sanity in installation [NW-NG-0 2012-09-24 17:50:09]: Step 19 of 20 Unmounting and cleaning up temporary file systems [NW-NG-0 2012-09-24 17:50:12]: Step 20 of 20 Setting da0s2 as new active partition [NW-NG-0 2012-09-24 17:50:23]: ------- P6136-C ------- [NW-NG-0 2012-09-24 17:50:23]: Step 1 of 20 Creating temporary file system [NW-NG-0 2012-09-24 17:50:23]: Step 2 of 20 Determining installation source [NW-NG-0 2012-09-24 17:50:23]: Step 3 of 20 Processing format options [NW-NG-0 2012-09-24 17:50:23]: Step 4 of 20 Determining installation slice [NW-NG-0 2012-09-24 17:50:23]: Step 5 of 20 Creating and labeling new slices [NW-NG-0 2012-09-24 17:50:23]: Step 6 of 20 Create and mount new file system [NW-NG-0 2012-09-24 17:50:23]: Step 7 of 20 Getting OS bundles [NW-NG-0 2012-09-24 17:50:23]: Step 8 of 20 Updating recovery media [NW-NG-0 2012-09-24 17:50:23]: Step 9 of 20 Extracting incoming image [NW-NG-0 2012-09-24 17:50:23]: Step 10 of 20 Unpacking OS packages [NW-NG-0 2012-09-24 17:50:23]: Step 11 of 20 Mounting jbase package [NW-NG-0 2012-09-24 17:50:23]: Step 12 of 20 Creating base OS symbolic links [NW-NG-0 2012-09-24 17:50:23]: Step 13 of 20 Creating fstab [NW-NG-0 2012-09-24 17:50:23]: Step 14 of 20 Creating new system files [NW-NG-0 2012-09-24 17:50:23]: Step 15 of 20 Adding jbundle package [NW-NG-0 2012-09-24 17:50:23]: Step 16 of 20 Backing up system data [NW-NG-0 2012-09-24 17:50:23]: Step 17 of 20 Setting up shared partition data [NW-NG-0 2012-09-24 17:50:23]: Step 18 of 20 Checking package sanity in installation [NW-NG-0 2012-09-24 17:50:23]: Step 19 of 20 Unmounting and cleaning up temporary file systems [NW-NG-0 2012-09-24 17:50:23]: Step 20 of 20 Setting da0s2 as new active partition [NW-NG-0 2012-09-24 17:50:27]: Step 1 of 20 Creating temporary file system [NW-NG-0 2012-09-24 17:50:27]: Step 2 of 20 Determining installation source [NW-NG-0 2012-09-24 17:50:27]: Step 3 of 20 Processing format options [NW-NG-0 2012-09-24 17:50:27]: Step 4 of 20 Determining installation slice [NW-NG-0 2012-09-24 17:50:27]: Step 5 of 20 Creating and labeling new slices [NW-NG-0 2012-09-24 17:50:27]: Step 6 of 20 Create and mount new file system [NW-NG-0 2012-09-24 17:50:27]: Step 7 of 20 Getting OS bundles [NW-NG-0 2012-09-24 17:50:27]: Step 8 of 20 Updating recovery media [NW-NG-0 2012-09-24 17:50:27]: Step 9 of 20 Extracting incoming image [NW-NG-0 2012-09-24 17:50:27]: Step 10 of 20 Unpacking OS packages [NW-NG-0 2012-09-24 17:50:27]: Step 11 of 20 Mounting jbase package [NW-NG-0 2012-09-24 17:50:27]: Step 12 of 20 Creating base OS symbolic links [NW-NG-0 2012-09-24 17:50:27]: Step 13 of 20 Creating fstab [NW-NG-0 2012-09-24 17:50:27]: Step 14 of 20 Creating new system files [NW-NG-0 2012-09-24 17:50:27]: Step 15 of 20 Adding jbundle package [NW-NG-0 2012-09-24 17:50:27]: Step 16 of 20 Backing up system data [NW-NG-0 2012-09-24 17:50:27]: Step 17 of 20 Setting up shared partition data [NW-NG-0 2012-09-24 17:50:27]: Step 18 of 20 Checking package sanity in installation [NW-NG-0 2012-09-24 17:50:27]: Step 19 of 20 Unmounting and cleaning up temporary file systems [NW-NG-0 2012-09-24 17:50:27]: Step 20 of 20 Setting da0s2 as new active partition [NW-NG-0 2012-09-24 17:50:27]: Step 1 of 20 Creating temporary file system [NW-NG-0 2012-09-24 17:50:27]: Step 2 of 20 Determining installation source [NW-NG-0 2012-09-24 17:50:27]: Step 3 of 20 Processing format options [NW-NG-0 2012-09-24 17:50:27]: Step 4 of 20 Determining installation slice [NW-NG-0 2012-09-24 17:50:27]: Step 5 of 20 Creating and labeling new slices [NW-NG-0 2012-09-24 17:50:27]: Step 6 of 20 Create and mount new file system [NW-NG-0 2012-09-24 17:50:27]: Step 7 of 20 Getting OS bundles [NW-NG-0 2012-09-24 17:50:27]: Step 8 of 20 Updating recovery media [NW-NG-0 2012-09-24 17:50:27]: Step 9 of 20 Extracting incoming image [NW-NG-0 2012-09-24 17:50:27]: Step 10 of 20 Unpacking OS packages [NW-NG-0 2012-09-24 17:50:27]: Step 11 of 20 Mounting jbase package [NW-NG-0 2012-09-24 17:50:27]: Step 12 of 20 Creating base OS symbolic links [NW-NG-0 2012-09-24 17:50:27]: Step 13 of 20 Creating fstab [NW-NG-0 2012-09-24 17:50:27]: Step 14 of 20 Creating new system files [NW-NG-0 2012-09-24 17:50:27]: Step 15 of 20 Adding jbundle package [NW-NG-0 2012-09-24 17:50:27]: Step 16 of 20 Backing up system data [NW-NG-0 2012-09-24 17:50:27]: Step 17 of 20 Setting up shared partition data [NW-NG-0 2012-09-24 17:50:27]: Step 18 of 20 Checking package sanity in installation [NW-NG-0 2012-09-24 17:50:27]: Step 19 of 20 Unmounting and cleaning up temporary file systems [NW-NG-0 2012-09-24 17:50:27]: Step 20 of 20 Setting da0s2 as new active partition [NW-NG-0 2012-09-24 17:50:27]: Starting with package ftp://169.254.0.3/pub/images/12.2X50-D20.4/jinstall-dc-re.tgz [NW-NG-0 2012-09-24 17:50:27]: Retrieving package [NW-NG-0 2012-09-24 17:51:35]: Pushing bundle to re0 [NW-NG-0 2012-09-24 17:52:09]: re0: Validate package... [NW-NG-0 2012-09-24 17:53:56]: re1: Validate package... [NW-NG-0 2012-09-24 17:55:53]: Rebooting Backup RE [NW-NG-0 2012-09-24 17:59:56]: Initiating Chassis In-Service-Upgrade [NW-NG-0 2012-09-24 18:00:16]: Upgrading group: 2 fpc: 2 [NW-NG-0 2012-09-24 18:10:08]: Upgrade complete for group:2 [NW-NG-0 2012-09-24 18:10:08]: Upgrading group: 3 fpc: 3 [NW-NG-0 2012-09-24 18:19:58]: Upgrade complete for group:3 [NW-NG-0 2012-09-24 18:19:58]: Upgrading group: 4 fpc: 4 [NW-NG-0 2012-09-24 18:29:45]: Upgrade complete for group:4 [NW-NG-0 2012-09-24 18:29:45]: Upgrading group: 5 fpc: 5 [NW-NG-0 2012-09-24 18:39:32]: Upgrade complete for group:5 [NW-NG-0 2012-09-24 18:39:32]: Finished processing all upgrade groups, last group :5 [NW-NG-0 2012-09-24 18:39:37]: Preparing for Switchover [NW-NG-0 2012-09-24 18:39:54]: Switchover Completed [Status 2012-09-24 18:39:54]: Upgrade completed with 0 errors Success root@FSASYDBRDQFAB01> ...0-D20.4.rpm node-group RSNG01 Upgrading target(s): RSNG01 [RSNG01 2012-09-25 11:44:47]: Starting with package ftp://169.254.0.3/pub/images/12.2X50-D20.4/jinstall-qfx.tgz [RSNG01 2012-09-25 11:44:47]: Retrieving package [RSNG01 2012-09-25 11:46:55]: Pushing bundle to P6167-C [RSNG01 2012-09-25 11:47:27]: P6167-C: Validate package... [RSNG01 2012-09-25 11:53:38]: P6185-C: Validate package... [RSNG01 2012-09-25 11:54:16]: ------- P6167-C ------- [RSNG01 2012-09-25 11:54:53]: Step 1 of 20 Creating temporary file system [RSNG01 2012-09-25 11:54:53]: Step 2 of 20 Determining installation source [RSNG01 2012-09-25 11:54:54]: Step 3 of 20 Processing format options [RSNG01 2012-09-25 11:54:54]: Step 4 of 20 Determining installation slice [RSNG01 2012-09-25 11:54:55]: Step 5 of 20 Creating and labeling new slices [RSNG01 2012-09-25 11:54:55]: Step 6 of 20 Create and mount new file system [RSNG01 2012-09-25 11:55:03]: Step 7 of 20 Getting OS bundles [RSNG01 2012-09-25 11:55:03]: Step 8 of 20 Updating recovery media [RSNG01 2012-09-25 11:55:25]: Step 9 of 20 Extracting incoming image [RSNG01 2012-09-25 11:56:40]: Step 10 of 20 Unpacking OS packages [RSNG01 2012-09-25 11:56:45]: Step 11 of 20 Mounting jbase package [RSNG01 2012-09-25 11:57:09]: Step 12 of 20 Creating base OS symbolic links [RSNG01 2012-09-25 11:58:10]: Step 13 of 20 Creating fstab [RSNG01 2012-09-25 11:58:11]: Step 14 of 20 Creating new system files [RSNG01 2012-09-25 11:58:11]: Step 15 of 20 Adding jbundle package [RSNG01 2012-09-25 12:00:48]: Step 16 of 20 Backing up system data [RSNG01 2012-09-25 12:00:50]: Step 17 of 20 Setting up shared partition data [RSNG01 2012-09-25 12:00:50]: Step 18 of 20 Checking package sanity in installation [RSNG01 2012-09-25 12:00:50]: Step 19 of 20 Unmounting and cleaning up temporary file systems [RSNG01 2012-09-25 12:00:54]: Step 20 of 20 Setting da0s2 as new active partition [RSNG01 2012-09-25 12:01:05]: ------- P6185-C - master ------- [RSNG01 2012-09-25 12:01:05]: Step 1 of 20 Creating temporary file system [RSNG01 2012-09-25 12:01:05]: Step 2 of 20 Determining installation source [RSNG01 2012-09-25 12:01:05]: Step 3 of 20 Processing format options [RSNG01 2012-09-25 12:01:05]: Step 4 of 20 Determining installation slice [RSNG01 2012-09-25 12:01:05]: Step 5 of 20 Creating and labeling new slices [RSNG01 2012-09-25 12:01:05]: Step 6 of 20 Create and mount new file system [RSNG01 2012-09-25 12:01:05]: Step 7 of 20 Getting OS bundles [RSNG01 2012-09-25 12:01:05]: Step 8 of 20 Updating recovery media [RSNG01 2012-09-25 12:01:05]: Step 9 of 20 Extracting incoming image [RSNG01 2012-09-25 12:01:05]: Step 10 of 20 Unpacking OS packages [RSNG01 2012-09-25 12:01:05]: Step 11 of 20 Mounting jbase package [RSNG01 2012-09-25 12:01:05]: Step 12 of 20 Creating base OS symbolic links [RSNG01 2012-09-25 12:01:05]: Step 13 of 20 Creating fstab [RSNG01 2012-09-25 12:01:05]: Step 14 of 20 Creating new system files [RSNG01 2012-09-25 12:01:05]: Step 15 of 20 Adding jbundle package [RSNG01 2012-09-25 12:01:05]: Step 16 of 20 Backing up system data [RSNG01 2012-09-25 12:01:05]: Step 17 of 20 Setting up shared partition data [RSNG01 2012-09-25 12:01:05]: Step 18 of 20 Checking package sanity in installation [RSNG01 2012-09-25 12:01:05]: Step 19 of 20 Unmounting and cleaning up temporary file systems [RSNG01 2012-09-25 12:01:05]: Step 20 of 20 Setting da0s2 as new active partition [RSNG01 2012-09-25 12:01:51]: Rebooting Backup RE [RSNG01 2012-09-25 12:01:51]: ------- Rebooting P6167-C ------- [RSNG01 2012-09-25 12:08:49]: Initiating Chassis In-Service-Upgrade [RSNG01 2012-09-25 12:09:09]: Upgrading group: 0 fpc: 0 [RSNG01 2012-09-25 12:11:15]: Upgrade complete for group:0 [RSNG01 2012-09-25 12:11:15]: Upgrading group: 1 fpc: 1 [RSNG01 2012-09-25 12:13:20]: Upgrade complete for group:1 [RSNG01 2012-09-25 12:13:20]: Finished processing all upgrade groups, last group :1 [RSNG01 2012-09-25 12:13:24]: Preparing for Switchover [RSNG01 2012-09-25 12:14:15]: Switchover Completed [Status 2012-09-25 12:14:15]: Upgrade completed with 0 errors Success |
Conclusion
The NSSU QFabric upgrade is a very simple and well polished process. Apart from being very time consuming, it's great and I really like how it's been designed and implemented. It's quite verbose and keeps the operator well informed, which I like, loving knowing what is actually going on. I also like (some may argue this is bad) the automatic upgrade of each SCB on the Interconnects and each slice on the nodes, saving that extra step post upgrade, but does make rollback harder.
Well done Juniper, this is another great part of the QFabric Solution!
P.s. Just give me a ssh client and automatic system archival.