Category: juniper

IPv6 + Flow labels

Recently a teammate and I have come across a frame forwarding issue with ECMP on a hardware ASIC in a device I work on where the use of Flow labels are used in the ECMP hash. This was interesting as we found iperf was not setting the Flow label at all, unless you specify the -L option and due to this we saw TCP traffic taking different paths, contradictory to what we thought we had configured in our FIB and what we actually wanted.

This sparked interest in me then wondering how popular platforms set the IPv6 Flow label for the different protocols; that being, ICMPv6, TCP and UDP. The Flow label being at Layer 3, I would expect it used the same for each protocol, but I could not find literature to back this theory up. So I fired up Wireshark on Mac, Linux and Windows to find out what they do. Here are my results I found.

If you want to know more about what Flow Labels are I would reccomened the following links:

  • Wikipedia: https://en.wikipedia.org/wiki/IPv6_packet#Fixed_header
  • RFC: https://tools.ietf.org/html/rfc6437

Summary

With each protocol the client and the server maintained consistent Flow labels for the ‘session’ as expected, except for Windows with ICMPv6 Requests! Here Windows set the Flow label to 0 (0x00000000).

Tests Performed

To get my results I ran:

  • ping6 -c 2 us.cooperlees.com
    (ICMPv6)
    – ping -6 us.cooperlees.com on Windows
  • ssh -6 us.cooperlees.com
    (TCP)
    – Used putty on Windows
  • Raw NTP UDP Query
    Python 3 Code: https://pastebin.com/RDBRqG0G
    (UDP)

Linux

Test Distro: Ubuntu 18.04
Test Kernel: 4.15.0-23-generic

ICMPv6
– Different Flow label, but consistent for the 2 ping packets on each ICMPv6 Type 128/129 packet from sender and receiver

TCP
– Different Flow label for sender and receiver but consistent across the SSH connection.

UDP
– Different Flow label for sender and receiver for each UDP packet as expected.

Mac OS X

Test Version: 10.13.6 17G65
Test Kernel: Darwin Kernel Version 17.7.0

ICMPv6
– Different Flow label, but consistent for the 2 ping packets on each ICMPv6 Type 128/129 packet from sender and receiver

TCP
– Different Flow label for sender and receiver but consistent across the SSH connection.

UDP
– Different Flow label for sender and receiver for each UDP packet as expected.

Windows

Test Version: Microsoft Windows [Version 10.0.16299.371]

ICMPv6
– Windows sets the ICMPv6 Type 128 (request) IPv6 Flow label to 0x00000000!
(I also noticed different DSCP for traffic class)

TCP
– Different Flow label for sender and receiver but consistent across the SSH connection.

UDP
– Different Flow label for sender and receiver for each UDP packet as expected.

IPv6 Tacacs+ Support (tac_plus)

Recently @ Facebook we found that we required IPv6 access to TACACS for auth (AAA) for the majority of our production Network Equipment. Tacacs+ (tac_plus) is an old daemon released by Cisco in the late 90s. It still works (even at our scale) and the config was doing what we required, so it was decided that we should add IPv6 Support to it to move forwards until we no longer require TACACS for authentication, authorization and accounting.

IPv6 has been added in true dirty 90s C code style via pre-processor macros. The source is publicly available via a GitHub Repository.

This version is based off F4.0.4.19 with the following patches (full history can be seen in the Git Repository):

  • Logging modifications
  • PAM Support
  • MD5 support
  • IPv6 (AF_INET6) Socket Listening

Readme.md has most of the information you require to build the software and I have included RPM .spec files (that have been tested on CentOS 6). The specs generate two RPMS with tacacs+6 relying on the tacacs+ rpm to be installed for libraries and man pages.

RPMS Build on CentOS 6.5 x86_64 + SRC rpms avaliable here: https://cooperlees.com/rpms/

Usage Tips:

  • Do not add listen directives into tac_plus.conf so that each daemon can load the same conf file (for consistency)
  • Logging:
    • /var/log/tac_plus.acct and tac_plus6.acct are where accounting information will go (as well as syslog) – Logrotate time …
    • /var/log/tac_plus.log and tac_plus6.log is where default debug logs will go
  • Configure syslog to send the LOG_LOCAL3 somewhere useful (this will get both tac_plus and tac_plus6 log information)
  • Pid Files will live in /var/run/tac_plus.pid.0.0.0.0 and tac_plus6.pid.::
  • The RPM does not /sbin/chkconfig –add or enable, so be sure to enable the version of tac_plus you require.

Tested Support on Vendor Hardware

  • Arista EoS (4.13.3F): need to use ‘ipv6 host name ::1’ as TACACS conf can’t handle raw IPv6 addresses (lame) 
  • Cisco NXOS (6.0(2)U2(4) [build 6.0(2)U2(3.6)]):
    feature tacacs+
    tacacs-server key 7 “c00p3rIstheMan”
    tacacs-server host a:cafe::1
    tacacs-server host b:b00c::2
    aaa group server tacacs+ TACACS
    server a:cafe::1
    server b:b00c::2
    source-interface Vlan2001 (ensure what IP request will come from)
  • Juniper: >= Junos 13.3R2.7 required for IPv6 Tacacs (Tested on MX)

I know it’s old school code but please feel free to submit bug patches / enhancements. This should allow us to keep this beast running until we can deprecate it’s need …

30 Levels of NAT Lab #2 – Juniper SRX100s

Well, I had the chance again to play with lots of Firewalls, so I did. A customer had ordered > than 30 SRX100s for clustered branch deployments so I took the opportunity to ask for permission to pull 30 of them out of boxes and reproduce my 30 levels of NAT lab. It’s never the same doing it alone so I put the word out to some nerd mates and got Mr Aijay Adams (@aijayadams) and Master Mitch Hewes (@mitcdh) to tag along and enjoy the extremly draining NAT filled day. The day included unboxing, ‘racking’ and cabling, configuring and then packing it all up. It was a long day, especially since we tried to get routing instances to NAT 10 times on each SRX … We were not successful, it seems LT interfaces are not NAT friendly (which is good, cause it’s stupid and I hate NAT).

Here is the logical layout:

natLab2_juniper_30.png

Of course, for good measure, a quick video of our fun 🙂

So we set the SRX’s up identically to the PIX501s

nat_lab_2.jpg
(not as neat – Time didn’t allow that :))

The results were similiar to the Cisco’s latency wise, but throughput was what I really expected originally (before I did the Cisco NAT LAB), the SRX100s were able to achieve the full 100mbit through the 30 levels of NAT.

After getting the same level of NAT achieved as the Cisco lab we set out to better it, by using routing-instances and lt interfaces, but it seems NAT from an lt-* interfaces is not supported in Junos (all be it a stupid requirement, still handy to know). If someone can see what I did wrong in the config below I would love to know. We had to call it quits after many hours trying to get the routing-instances to work, but we were so close!
P.S. I know I could of used all the physicals, I did not have enough patch cables !

Here is a screenshot of a traceroute of out NATHELL

With no throughput:

traceroute_no_throughput.png

With 100mbit of throughput:

tracroute_troughput.png

Nerdy Setup Details

1 NAT per Box Config:

[bash]
set system host-name NATLAB0
set system domain-name cooperlees.com
set system root-authentication encrypted-password “lab123”
set system name-server 192.168.83.6
set system name-server 192.168.83.5
set system name-server 8.8.8.8
set system services ssh
set system services dhcp pool 10.0.0.0/24 address-range low 10.0.0.10
set system services dhcp pool 10.0.0.0/24 address-range high 10.0.0.100
set system services dhcp pool 10.0.0.0/24 name-server 192.168.83.5
set system services dhcp pool 10.0.0.0/24 name-server 192.168.83.6
set system services dhcp pool 10.0.0.0/24 name-server 8.8.8.8
set system services dhcp pool 10.0.0.0/24 domain-search cooperlees.com
set system services dhcp pool 10.0.0.0/24 router 10.0.0.1
set interfaces interface-range ACCESS member-range fe-0/0/0 to fe-0/0/6
set interfaces interface-range ACCESS unit 0 family ethernet-switching
set interfaces fe-0/0/7 unit 0 family inet address 10.1.0.2/24
set interfaces vlan unit 0 family inet address 10.0.0.1/24
set routing-options static route 0.0.0.0/0 next-hop 10.1.0.1
set security nat source rule-set Outbound-NAT from zone NATHELL
set security nat source rule-set Outbound-NAT to zone LESSNAT
set security nat source rule-set Outbound-NAT rule egress-int-bitch match destination-address 0.0.0.0/0
set security nat source rule-set Outbound-NAT rule egress-int-bitch then source-nat interface
set security nat destination pool WEBSERVER address 10.0.0.2/32
set security nat destination rule-set Internet-PAT from zone LESSNAT
set security nat destination rule-set Internet-PAT rule WEBSERVER-PAT match destination-address 10.1.0.2/32
set security nat destination rule-set Internet-PAT rule WEBSERVER-PAT match destination-port 80
set security nat destination rule-set Internet-PAT rule WEBSERVER-PAT then destination-nat pool WEBSERVER
set security policies from-zone NATHELL to-zone LESSNAT policy allow-all-bitch match source-address any
set security policies from-zone NATHELL to-zone LESSNAT policy allow-all-bitch match destination-address any
set security policies from-zone NATHELL to-zone LESSNAT policy allow-all-bitch match application any
set security policies from-zone NATHELL to-zone LESSNAT policy allow-all-bitch then permit
set security policies from-zone NATHELL to-zone LESSNAT policy allow-all-bitch then log session-init
set security zones security-zone NATHELL host-inbound-traffic system-services all
set security zones security-zone NATHELL host-inbound-traffic protocols all
set security zones security-zone NATHELL interfaces vlan.0
set security zones security-zone LESSNAT host-inbound-traffic system-services all
set security zones security-zone LESSNAT host-inbound-traffic protocols all
set security zones security-zone LESSNAT interfaces fe-0/0/7.0
set vlans default l3-interface vlan.0
[/bash]

10 NATs per Box Attempt:

This config did not work with the NAT’ing between RI’s over the LT interfaces. The goal was:

10-routing-instances.png

[bash]
set system host-name NATLAB1
set system domain-name cooperlees.com
set system root-authentication encrypted-password “$1$fU1Lb028$c/LeEFORggONDEgKovRyj.”
set system name-server 192.168.83.6
set system name-server 192.168.83.5
set system name-server 8.8.8.8
set system services ssh
set system services dhcp pool 10.1.0.0/24 address-range low 10.1.0.10
set system services dhcp pool 10.1.0.0/24 address-range high 10.1.0.100
set system services dhcp pool 10.1.0.0/24 name-server 192.168.83.5
set system services dhcp pool 10.1.0.0/24 name-server 192.168.83.6
set system services dhcp pool 10.1.0.0/24 name-server 8.8.8.8
set system services dhcp pool 10.1.0.0/24 domain-search cooperlees.com
set system services dhcp pool 10.1.0.0/24 router 10.1.0.1
set interfaces interface-range ACCESS member-range fe-0/0/0 to fe-0/0/6
set interfaces interface-range ACCESS unit 0 family ethernet-switching
set interfaces lt-0/0/0 unit 0 encapsulation ethernet
set interfaces lt-0/0/0 unit 0 peer-unit 1
set interfaces lt-0/0/0 unit 0 family inet address 10.1.1.1/24
set interfaces lt-0/0/0 unit 1 encapsulation ethernet
set interfaces lt-0/0/0 unit 1 peer-unit 0
set interfaces lt-0/0/0 unit 1 family inet address 10.1.1.2/24
set interfaces lt-0/0/0 unit 2 encapsulation ethernet
set interfaces lt-0/0/0 unit 2 peer-unit 3
set interfaces lt-0/0/0 unit 2 family inet address 10.1.2.1/24
set interfaces lt-0/0/0 unit 3 encapsulation ethernet
set interfaces lt-0/0/0 unit 3 peer-unit 2
set interfaces lt-0/0/0 unit 3 family inet address 10.1.2.2/24
set interfaces lt-0/0/0 unit 4 encapsulation ethernet
set interfaces lt-0/0/0 unit 4 peer-unit 5
set interfaces lt-0/0/0 unit 4 family inet address 10.1.3.1/24
set interfaces lt-0/0/0 unit 5 encapsulation ethernet
set interfaces lt-0/0/0 unit 5 peer-unit 4
set interfaces lt-0/0/0 unit 5 family inet address 10.1.3.2/24
set interfaces lt-0/0/0 unit 6 encapsulation ethernet
set interfaces lt-0/0/0 unit 6 peer-unit 7
set interfaces lt-0/0/0 unit 6 family inet address 10.1.4.1/24
set interfaces lt-0/0/0 unit 7 encapsulation ethernet
set interfaces lt-0/0/0 unit 7 peer-unit 6
set interfaces lt-0/0/0 unit 7 family inet address 10.1.4.2/24
set interfaces lt-0/0/0 unit 8 encapsulation ethernet
set interfaces lt-0/0/0 unit 8 peer-unit 9
set interfaces lt-0/0/0 unit 8 family inet address 10.1.5.1/24
set interfaces lt-0/0/0 unit 9 encapsulation ethernet
set interfaces lt-0/0/0 unit 9 peer-unit 8
set interfaces lt-0/0/0 unit 9 family inet address 10.1.5.2/24
set interfaces lt-0/0/0 unit 10 encapsulation ethernet
set interfaces lt-0/0/0 unit 10 peer-unit 11
set interfaces lt-0/0/0 unit 10 family inet address 10.1.6.1/24
set interfaces lt-0/0/0 unit 11 encapsulation ethernet
set interfaces lt-0/0/0 unit 11 peer-unit 10
set interfaces lt-0/0/0 unit 11 family inet address 10.1.6.2/24
set interfaces lt-0/0/0 unit 12 encapsulation ethernet
set interfaces lt-0/0/0 unit 12 peer-unit 13
set interfaces lt-0/0/0 unit 12 family inet address 10.1.7.1/24
set interfaces lt-0/0/0 unit 13 encapsulation ethernet
set interfaces lt-0/0/0 unit 13 peer-unit 12
set interfaces lt-0/0/0 unit 13 family inet address 10.1.7.2/24
set interfaces lt-0/0/0 unit 14 encapsulation ethernet
set interfaces lt-0/0/0 unit 14 peer-unit 15
set interfaces lt-0/0/0 unit 14 family inet address 10.1.8.1/24
set interfaces lt-0/0/0 unit 15 encapsulation ethernet
set interfaces lt-0/0/0 unit 15 peer-unit 14
set interfaces lt-0/0/0 unit 15 family inet address 10.1.8.2/24
set interfaces lt-0/0/0 unit 16 encapsulation ethernet
set interfaces lt-0/0/0 unit 16 peer-unit 17
set interfaces lt-0/0/0 unit 16 family inet address 10.1.9.1/24
set interfaces lt-0/0/0 unit 17 encapsulation ethernet
set interfaces lt-0/0/0 unit 17 peer-unit 16
set interfaces lt-0/0/0 unit 17 family inet address 10.1.9.2/24
set interfaces fe-0/0/7 unit 0 family inet address 10.2.0.2/24
set interfaces vlan unit 0 family inet address 10.1.0.1/24
set routing-options static route 0.0.0.0/0 next-hop 10.2.0.1
set security nat source rule-set Outbound-NAT from interface lt-0/0/0.17
set security nat source rule-set Outbound-NAT to interface fe-0/0/7.0
set security nat source rule-set Outbound-NAT rule egress-int-bitch match destination-address 0.0.0.0/0
set security nat source rule-set Outbound-NAT rule egress-int-bitch then source-nat interface
set security nat source rule-set th3 from zone transitHELL-2
set security nat source rule-set th3 to zone transitHELL-3
set security nat source rule-set th3 rule natAll-1 match destination-address 0.0.0.0/0
set security nat source rule-set th3 rule natAll-1 then source-nat interface
set security nat source rule-set th4 from interface lt-0/0/0.4
set security nat source rule-set th4 to interface lt-0/0/0.5
set security nat source rule-set th4 rule natAll-2 match destination-address 0.0.0.0/0
set security nat source rule-set th4 rule natAll-2 then source-nat interface
set security nat source rule-set th5 from interface lt-0/0/0.6
set security nat source rule-set th5 to interface lt-0/0/0.7
set security nat source rule-set th5 rule natAll-3 match destination-address 0.0.0.0/0
set security nat source rule-set th5 rule natAll-3 then source-nat interface
set security nat source rule-set th6 from interface lt-0/0/0.8
set security nat source rule-set th6 to interface lt-0/0/0.9
set security nat source rule-set th6 rule natAll-4 match destination-address 0.0.0.0/0
set security nat source rule-set th6 rule natAll-4 then source-nat interface
set security nat source rule-set th7 from interface lt-0/0/0.10
set security nat source rule-set th7 to interface lt-0/0/0.11
set security nat source rule-set th7 rule natAll-5 match destination-address 0.0.0.0/0
set security nat source rule-set th7 rule natAll-5 then source-nat interface
set security nat source rule-set th8 from interface lt-0/0/0.12
set security nat source rule-set th8 to interface lt-0/0/0.13
set security nat source rule-set th8 rule natAll-6 match destination-address 0.0.0.0/0
set security nat source rule-set th8 rule natAll-6 then source-nat interface
set security nat source rule-set Inbound-NAT from interface vlan.0
set security nat source rule-set Inbound-NAT to interface lt-0/0/0.0
set security nat source rule-set Inbound-NAT rule natAll-0 match destination-address 0.0.0.0/0
set security nat source rule-set Inbound-NAT rule natAll-0 then source-nat interface
set security nat source rule-set th9 from interface lt-0/0/0.14
set security nat source rule-set th9 to interface lt-0/0/0.15
set security nat source rule-set th9 rule natAll-7 match destination-address 0.0.0.0/0
set security nat source rule-set th9 rule natAll-7 then source-nat interface
set security nat source rule-set th10 from interface lt-0/0/0.16
set security nat source rule-set th10 to interface lt-0/0/0.17
set security nat source rule-set th10 rule natAll-8 match destination-address 0.0.0.0/0
set security nat source rule-set th10 rule natAll-8 then source-nat interface
set security nat destination pool WEBSERVER address 10.1.0.10/32
set security nat destination rule-set Internet-PAT from zone LESSNAT
set security nat destination rule-set Internet-PAT rule WEBSERVER-PAT match destination-address 10.2.0.2/32
set security nat destination rule-set Internet-PAT rule WEBSERVER-PAT match destination-port 80
set security nat destination rule-set Internet-PAT rule WEBSERVER-PAT then destination-nat pool WEBSERVER
set security policies from-zone NATHELL to-zone LESSNAT policy allow-all-bitch match source-address any
set security policies from-zone NATHELL to-zone LESSNAT policy allow-all-bitch match destination-address any
set security policies from-zone NATHELL to-zone LESSNAT policy allow-all-bitch match application any
set security policies from-zone NATHELL to-zone LESSNAT policy allow-all-bitch then permit
set security policies from-zone NATHELL to-zone LESSNAT policy allow-all-bitch then log session-init
set security policies from-zone LESSNAT to-zone NATHELL policy allow-webserver-traffic match source-address any
set security policies from-zone LESSNAT to-zone NATHELL policy allow-webserver-traffic match destination-address any
set security policies from-zone LESSNAT to-zone NATHELL policy allow-webserver-traffic match application junos-icmp-all
set security policies from-zone LESSNAT to-zone NATHELL policy allow-webserver-traffic match application junos-http
set security policies from-zone LESSNAT to-zone NATHELL policy allow-webserver-traffic match application junos-ssh
set security policies from-zone LESSNAT to-zone NATHELL policy allow-webserver-traffic then permit
set security policies from-zone LESSNAT to-zone NATHELL policy allow-webserver-traffic then log session-init
set security policies from-zone NATHELL to-zone transitHELL-2 policy NATHELL-to-th2 match source-address any
set security policies from-zone NATHELL to-zone transitHELL-2 policy NATHELL-to-th2 match destination-address any
set security policies from-zone NATHELL to-zone transitHELL-2 policy NATHELL-to-th2 match application any
set security policies from-zone NATHELL to-zone transitHELL-2 policy NATHELL-to-th2 then permit
set security policies from-zone transitHELL-2 to-zone transitHELL-3 policy allow-transit-2 match source-address any
set security policies from-zone transitHELL-2 to-zone transitHELL-3 policy allow-transit-2 match destination-address any
set security policies from-zone transitHELL-2 to-zone transitHELL-3 policy allow-transit-2 match application any
set security policies from-zone transitHELL-2 to-zone transitHELL-3 policy allow-transit-2 then permit
set security policies from-zone transitHELL-3 to-zone transitHELL-4 policy allow-transit-4 match source-address any
set security policies from-zone transitHELL-3 to-zone transitHELL-4 policy allow-transit-4 match destination-address any
set security policies from-zone transitHELL-3 to-zone transitHELL-4 policy allow-transit-4 match application any
set security policies from-zone transitHELL-3 to-zone transitHELL-4 policy allow-transit-4 then permit
set security policies from-zone transitHELL-4 to-zone transitHELL-5 policy allow-transit-5 match source-address any
set security policies from-zone transitHELL-4 to-zone transitHELL-5 policy allow-transit-5 match destination-address any
set security policies from-zone transitHELL-4 to-zone transitHELL-5 policy allow-transit-5 match application any
set security policies from-zone transitHELL-4 to-zone transitHELL-5 policy allow-transit-5 then permit
set security policies from-zone transitHELL-5 to-zone transitHELL-6 policy allow-transit-6 match source-address any
set security policies from-zone transitHELL-5 to-zone transitHELL-6 policy allow-transit-6 match destination-address any
set security policies from-zone transitHELL-5 to-zone transitHELL-6 policy allow-transit-6 match application any
set security policies from-zone transitHELL-5 to-zone transitHELL-6 policy allow-transit-6 then permit
set security policies from-zone transitHELL-6 to-zone transitHELL-7 policy allow-transit-7 match source-address any
set security policies from-zone transitHELL-6 to-zone transitHELL-7 policy allow-transit-7 match destination-address any
set security policies from-zone transitHELL-6 to-zone transitHELL-7 policy allow-transit-7 match application any
set security policies from-zone transitHELL-6 to-zone transitHELL-7 policy allow-transit-7 then permit
set security policies from-zone transitHELL-7 to-zone transitHELL-8 policy allow-transit-8 match source-address any
set security policies from-zone transitHELL-7 to-zone transitHELL-8 policy allow-transit-8 match destination-address any
set security policies from-zone transitHELL-7 to-zone transitHELL-8 policy allow-transit-8 match application any
set security policies from-zone transitHELL-7 to-zone transitHELL-8 policy allow-transit-8 then permit
set security policies from-zone transitHELL-8 to-zone transitHELL-9 policy allow-transit-9 match source-address any
set security policies from-zone transitHELL-8 to-zone transitHELL-9 policy allow-transit-9 match destination-address any
set security policies from-zone transitHELL-8 to-zone transitHELL-9 policy allow-transit-9 match application any
set security policies from-zone transitHELL-8 to-zone transitHELL-9 policy allow-transit-9 then permit
set security policies from-zone transitHELL-9 to-zone LESSNAT policy 9-to-LESSNAT match source-address any
set security policies from-zone transitHELL-9 to-zone LESSNAT policy 9-to-LESSNAT match destination-address any
set security policies from-zone transitHELL-9 to-zone LESSNAT policy 9-to-LESSNAT match application any
set security policies from-zone transitHELL-9 to-zone LESSNAT policy 9-to-LESSNAT then permit
set security policies default-policy permit-all
set security zones security-zone NATHELL host-inbound-traffic system-services all
set security zones security-zone NATHELL host-inbound-traffic protocols all
set security zones security-zone NATHELL interfaces vlan.0
set security zones security-zone NATHELL interfaces lt-0/0/0.0
set security zones security-zone LESSNAT host-inbound-traffic system-services all
set security zones security-zone LESSNAT host-inbound-traffic protocols all
set security zones security-zone LESSNAT interfaces fe-0/0/7.0
set security zones security-zone LESSNAT interfaces lt-0/0/0.17
set security zones security-zone transitHELL-2 host-inbound-traffic system-services all
set security zones security-zone transitHELL-2 host-inbound-traffic protocols all
set security zones security-zone transitHELL-2 interfaces lt-0/0/0.1
set security zones security-zone transitHELL-2 interfaces lt-0/0/0.2
set security zones security-zone transitHELL-3 host-inbound-traffic system-services all
set security zones security-zone transitHELL-3 host-inbound-traffic protocols all
set security zones security-zone transitHELL-3 interfaces lt-0/0/0.3
set security zones security-zone transitHELL-3 interfaces lt-0/0/0.4
set security zones security-zone transitHELL-4 host-inbound-traffic system-services all
set security zones security-zone transitHELL-4 host-inbound-traffic protocols all
set security zones security-zone transitHELL-4 interfaces lt-0/0/0.5
set security zones security-zone transitHELL-4 interfaces lt-0/0/0.6
set security zones security-zone transitHELL-5 host-inbound-traffic system-services all
set security zones security-zone transitHELL-5 host-inbound-traffic protocols all
set security zones security-zone transitHELL-5 interfaces lt-0/0/0.7
set security zones security-zone transitHELL-5 interfaces lt-0/0/0.8
set security zones security-zone transitHELL-6 host-inbound-traffic system-services all
set security zones security-zone transitHELL-6 host-inbound-traffic protocols all
set security zones security-zone transitHELL-6 interfaces lt-0/0/0.9
set security zones security-zone transitHELL-6 interfaces lt-0/0/0.10
set security zones security-zone transitHELL-7 host-inbound-traffic system-services all
set security zones security-zone transitHELL-7 host-inbound-traffic protocols all
set security zones security-zone transitHELL-7 interfaces lt-0/0/0.11
set security zones security-zone transitHELL-7 interfaces lt-0/0/0.12
set security zones security-zone transitHELL-8 host-inbound-traffic system-services all
set security zones security-zone transitHELL-8 host-inbound-traffic protocols all
set security zones security-zone transitHELL-8 interfaces lt-0/0/0.13
set security zones security-zone transitHELL-8 interfaces lt-0/0/0.14
set security zones security-zone transitHELL-9 host-inbound-traffic system-services all
set security zones security-zone transitHELL-9 host-inbound-traffic protocols all
set security zones security-zone transitHELL-9 interfaces lt-0/0/0.15
set security zones security-zone transitHELL-9 interfaces lt-0/0/0.16
set routing-instances 1 instance-type virtual-router
set routing-instances 1 interface lt-0/0/0.0
set routing-instances 1 interface vlan.0
set routing-instances 1 routing-options static route 0.0.0.0/0 next-hop 10.1.1.2
set routing-instances 10 instance-type virtual-router
set routing-instances 10 interface lt-0/0/0.17
set routing-instances 10 interface fe-0/0/7.0
set routing-instances 10 routing-options static route 0.0.0.0/0 next-hop 10.2.0.1
set routing-instances 2 instance-type virtual-router
set routing-instances 2 interface lt-0/0/0.1
set routing-instances 2 interface lt-0/0/0.2
set routing-instances 2 routing-options static route 0.0.0.0/0 next-hop 10.1.2.2
set routing-instances 3 instance-type virtual-router
set routing-instances 3 interface lt-0/0/0.3
set routing-instances 3 interface lt-0/0/0.4
set routing-instances 3 routing-options static route 0.0.0.0/0 next-hop 10.1.3.2
set routing-instances 4 instance-type virtual-router
set routing-instances 4 interface lt-0/0/0.5
set routing-instances 4 interface lt-0/0/0.6
set routing-instances 4 routing-options static route 0.0.0.0/0 next-hop 10.1.4.2
set routing-instances 5 instance-type virtual-router
set routing-instances 5 interface lt-0/0/0.7
set routing-instances 5 interface lt-0/0/0.8
set routing-instances 5 routing-options static route 0.0.0.0/0 next-hop 10.1.5.2
set routing-instances 6 instance-type virtual-router
set routing-instances 6 interface lt-0/0/0.9
set routing-instances 6 interface lt-0/0/0.10
set routing-instances 6 routing-options static route 0.0.0.0/0 next-hop 10.1.6.2
set routing-instances 7 instance-type virtual-router
set routing-instances 7 interface lt-0/0/0.11
set routing-instances 7 interface lt-0/0/0.12
set routing-instances 7 routing-options static route 0.0.0.0/0 next-hop 10.1.7.2
set routing-instances 8 instance-type virtual-router
set routing-instances 8 interface lt-0/0/0.13
set routing-instances 8 interface lt-0/0/0.14
set routing-instances 8 routing-options static route 0.0.0.0/0 next-hop 10.1.8.2
set routing-instances 9 instance-type virtual-router
set routing-instances 9 interface lt-0/0/0.15
set routing-instances 9 interface lt-0/0/0.16
set routing-instances 9 routing-options static route 0.0.0.0/0 next-hop 10.1.9.2
set vlans default l3-interface vlan.0
[/bash]

Conf Generation Shell Script:

[bash]
#!/bin/bash

for i in $(seq 0 30)
do
echo -n “Doing $i …”

N1=$(expr $i + 1)

cp template.txt $i.conf

# Replace N1
sed -i s/#N1/$N1/g $i.conf

# Replace N
sed -i s/#N/$i/g $i.conf

echo ” done”
done
[/bash]

Until we NAT again …

RANCID with Junos Read-Only User

Here is the setting for a Junos device to create a user with read only privileges to allow RANCID to work.

[plain]
set system login class RANCID permissions access
set system login class RANCID permissions admin
set system login class RANCID permissions firewall
set system login class RANCID permissions flow-tap
set system login class RANCID permissions interface
set system login class RANCID permissions network
set system login class RANCID permissions routing
set system login class RANCID permissions secret
set system login class RANCID permissions security
set system login class RANCID permissions snmp
set system login class RANCID permissions storage
set system login class RANCID permissions system
set system login class RANCID permissions trace
set system login class RANCID permissions view
set system login class RANCID permissions view-configuration

set system login user rancid full-name RANCID
set system login user rancid class RANCID
set system login user rancid authentication encrypted-password “xxx”
[/plain]

Updating Juniper QFabric

The follow post shows output obtained and the  upgrade process performed recently on a clients QFabric system. This output was captured updating from 12.2X30 to 12.2X50 Junos release via a ‘Non Stop Services Upgrade’ (NSSU) method. This method basically is a very conservative approach updating redundant components one at a time.

The overall process is:

  1. Upgrade Director Group
  2. Upgrade QFabric Interconnects
  3. Upgrade each node group
    1. Network Node group (NW-NG-01)
    2. Each redundant server node group (RSNG)
    3. Each server node group (my client did not have any SNGs)

Before Upgrade Backup

All that is required to be backed up is the QFabric configuration file, everything else about the install is the QFabric standard and able to be restored using documented Juniper methods.

To backup the config log into the device and:

  1. Capture the output from ‘show configuration | no-more’

or

  1. ‘show configuration | save QFabric.conf’
    1. Remotely: scp username@x.x.x.x:/pbdata/packages/QFabric.conf

Upgrade Process with Output

Director Group Upgrade

Copy the RPM image to the director to /pbdata/packages. This process takes around 2 hours. We started at 7:15am and finished at 9:15am.

  1. scp FILE.rpm root@x.x.x.x:/pbdata/packages
  2. Log into the DG via the VIP and start the upgrade
  • request system software nonstop-upgrade director-group FILE.rpm
    • Junos looks in /pbdata/packages by default

Upgrade Output:

root@FSASYDBRDQFAB01> request system software nonstop-upgrade director-group jinstall-qfabric-12.2X50-D20.4.rpmValidating update package jinstall-qfabric-12.2X50-D20.4.rpmInstalling update package jinstall-qfabric-12.2X50-D20.4.rpmInstalling fabric images version 12.2X50-D20.4Performing cleanupPackage install completeInstalling update package jinstall-qfabric-12.2X50-D20.4.rpm on peer

Triggering Initial Stage of Fabric Manager Upgrade

Updating CCIF default image to 12.2X50-D20.4

Updating FM-0 to Junos version 12.2X50-D20.4

[Status   2012-09-24 14:43:37]: Fabric Manager: Upgrade Initial Stage started

[FM-0     2012-09-24 14:43:52]: Transferring FM-0 Mastership to LOCAL DG

[FM-0     2012-09-24 14:45:44]: Finished FM-0 Mastership switch

[NW-NG-0  2012-09-24 14:45:59]: Transferring NW-NG-0 Mastership to LOCAL DG

[NW-NG-0  2012-09-24 14:47:22]: Finished NW-NG-0 Mastership switch

[FM-0     2012-09-24 14:48:10]: Retrieving package

[FM-0     2012-09-24 14:49:13]: Retrieving package

[FM-0     2012-09-24 14:50:15]: Pushing bundle to re0

[Status   2012-09-24 14:52:03]: Load completed with 0 errors

[Status   2012-09-24 14:52:03]: Reboot is required to complete upgrade

[Status   2012-09-24 14:52:04]: Trying to Connect to Node: FM-0

[Status   2012-09-24 14:52:19]: Rebooting FM-0

[FM-0     2012-09-24 14:52:19]: Waiting for FM-0 to terminate

Starting Peer upgrade

Initiating rolling upgrade of Director peer:  version 12.2X50-D20.4

Inform CCIF regarding rolling upgrade

[Peer Update Status]: Validating install package jinstall-qfabric-12.2X50-D20.4.rpm

[Peer Update Status]: jinstall-qfabric-12.2X50.D20.4-4

[Peer Update Status]: Cleaning up node for rolling phase one upgrade

[Peer Update Status]: Director group upgrade complete

[Peer Update Status]: COMPLETED

[Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade

[Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade

[Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade

[Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade

[Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade

[Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade

[Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade

[Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade

[Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade

[Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade

[Peer Update Status]: Waiting for peer to return after reboot and start phase one of rolling upgrade

[Peer Update Status]: Waiting for peer to return after reboot and start phase one of rolling upgrade

[Peer Update Status]: Waiting for peer to return after reboot and start phase one of rolling upgrade

[Peer Update Status]: Waiting for peer to return after reboot and start phase one of rolling upgrade

[Peer Update Status]: Waiting for peer to return after reboot and start phase one of rolling upgrade

[Peer Update Status]: Waiting for peer to return after reboot and start phase one of rolling upgrade

[Peer Update Status]: Waiting for peer to return after reboot and start phase one of rolling upgrade

[Peer Update Status]: Waiting for peer to return after reboot and start phase one of rolling upgrade

[Peer Update Status]: Waiting for peer to complete phase one of rolling upgrade

[Peer Update Status]: Waiting for peer to complete phase one of rolling upgrade

[Peer Update Status]: Peer completed phase one of rolling upgrade

Setting peer DG node as the master SFC

Delaying start of local upgrade to allow peer services time to initialize [15 minutes]

Delaying start of local upgrade to allow peer services time to initialize [15 minutes]

Delaying start of local upgrade to allow peer services time to initialize [12 minutes]

Delaying start of local upgrade to allow peer services time to initialize [9 minutes]

Delaying start of local upgrade to allow peer services time to initialize [6 minutes]

Delaying start of local upgrade to allow peer services time to initialize [3 minutes]

[Peer Update Status]: Check for VMs on dg0

Triggering Final Stage of Fabric Manager Upgrade:

Updating FM-0 to Junos version 12.2X50-D20.4

[Status   2012-09-24 15:33:31]: Fabric Manager: Upgrade Final Stage started

[NW-NG-0  2012-09-24 15:33:45]: Transferring NW-NG-0 Mastership to REMOTE DG

[NW-NG-0  2012-09-24 15:35:08]: Finished NW-NG-0 Mastership switch

[Status   2012-09-24 15:35:08]: Upgrading FM-0 VM on worker DG to 12.2X50-D20.4

[DRE-0    2012-09-24 15:36:09]: Retrieving package

[DRE-0    2012-09-24 15:37:02]: ——- re0: ——-

[Status   2012-09-24 15:38:28]: Load completed with 0 errors

[Status   2012-09-24 15:38:28]: Reboot is required to complete upgrade

[DRE-0    2012-09-24 15:38:34]: Waiting for DRE-0 to terminate

[DRE-0    2012-09-24 15:38:46]: Waiting for DRE-0 to come back

[DRE-0    2012-09-24 15:42:00]: Running Uptime Test for DRE-0

[DRE-0    2012-09-24 15:42:06]: Uptime Test for DRE-0 Passed

[Status   2012-09-24 15:42:06]: DRE-0 Booted successfully

Performing post install shutdown and cleanup

Broadcast message from root (Mon Sep 24 15:42:07 2012):

The system is going down for reboot NOW!

Director group upgrade complete

Interconnect Upgrade

This process takes around an hour. It will upgrade Junos on each System Control Board (SCB) partition grabbing the code automatically via the FTP running on the active Director Group member. We observed roughly that time starting at 7:15am and finished at 9:15am.

  1. From the DG CLI initiate the config:
  • request system software nonstop-upgrade fabric FILE.rpm

Output:

[FC-0     2012-09-24 16:22:17]: Retrieving package[FC-1     2012-09-24 16:22:18]: Retrieving package[IC-F7811 2012-09-24 16:22:39]: Retrieving package[IC-F7712 2012-09-24 16:22:41]: Retrieving package[FC-0     2012-09-24 16:23:14]: Validating on re0[FC-1     2012-09-24 16:23:18]: Validating on re0[IC-F7712 2012-09-24 16:23:57]: Pushing bundle to re1

[IC-F7811 2012-09-24 16:23:58]: Pushing bundle to re1

[IC-F7712 2012-09-24 16:24:47]: Validating on re1

[IC-F7811 2012-09-24 16:24:48]: Validating on re1

[FC-0     2012-09-24 16:25:02]: Done with validate on all chassis

[FC-0     2012-09-24 16:25:02]: ——- re0: ——-

[FC-1     2012-09-24 16:25:11]: Done with validate on all chassis

[FC-1     2012-09-24 16:25:11]: ——- re0: ——-

[IC-F7712 2012-09-24 16:29:51]: Validating on re0

[IC-F7811 2012-09-24 16:30:48]: Validating on re0

[IC-F7712 2012-09-24 16:34:10]: Done with validate on all chassis

[IC-F7712 2012-09-24 16:34:10]: ——- re1: ——-

[IC-F7811 2012-09-24 16:34:20]: Done with validate on all chassis

[IC-F7811 2012-09-24 16:34:20]: ——- re1: ——-

[IC-F7712 2012-09-24 16:34:55]: Step 1 of 20 Creating temporary file system

[IC-F7712 2012-09-24 16:34:55]: Step 2 of 20 Determining installation source

[IC-F7712 2012-09-24 16:34:55]: Step 3 of 20 Processing format options

[IC-F7712 2012-09-24 16:34:55]: Step 4 of 20 Determining installation slice

[IC-F7712 2012-09-24 16:34:56]: Step 5 of 20 Creating and labeling new slices

[IC-F7811 2012-09-24 16:34:56]: Step 1 of 20 Creating temporary file system

[IC-F7712 2012-09-24 16:34:56]: Step 6 of 20 Create and mount new file system

[IC-F7811 2012-09-24 16:34:57]: Step 2 of 20 Determining installation source

[IC-F7811 2012-09-24 16:34:57]: Step 3 of 20 Processing format options

[IC-F7811 2012-09-24 16:34:57]: Step 4 of 20 Determining installation slice

[IC-F7811 2012-09-24 16:34:58]: Step 5 of 20 Creating and labeling new slices

[IC-F7811 2012-09-24 16:34:58]: Step 6 of 20 Create and mount new file system

[IC-F7712 2012-09-24 16:35:04]: Step 7 of 20 Getting OS bundles

[IC-F7712 2012-09-24 16:35:04]: Step 8 of 20 Updating recovery media

[IC-F7811 2012-09-24 16:35:07]: Step 7 of 20 Getting OS bundles

[IC-F7811 2012-09-24 16:35:07]: Step 8 of 20 Updating recovery media

[IC-F7712 2012-09-24 16:35:27]: Step 9 of 20 Extracting incoming image

[IC-F7811 2012-09-24 16:35:30]: Step 9 of 20 Extracting incoming image

[IC-F7712 2012-09-24 16:36:38]: Step 10 of 20 Unpacking OS packages

[IC-F7712 2012-09-24 16:36:41]: Step 11 of 20 Mounting jbase package

[IC-F7811 2012-09-24 16:36:42]: Step 10 of 20 Unpacking OS packages

[IC-F7811 2012-09-24 16:36:45]: Step 11 of 20 Mounting jbase package

[IC-F7712 2012-09-24 16:37:05]: Step 12 of 20 Creating base OS symbolic links

[IC-F7811 2012-09-24 16:37:09]: Step 12 of 20 Creating base OS symbolic links

[IC-F7712 2012-09-24 16:38:03]: Step 13 of 20 Creating fstab

[IC-F7712 2012-09-24 16:38:03]: Step 14 of 20 Creating new system files

[IC-F7712 2012-09-24 16:38:04]: Step 15 of 20 Adding jbundle package

[IC-F7811 2012-09-24 16:38:07]: Step 13 of 20 Creating fstab

[IC-F7811 2012-09-24 16:38:07]: Step 14 of 20 Creating new system files

[IC-F7811 2012-09-24 16:38:07]: Step 15 of 20 Adding jbundle package

[IC-F7712 2012-09-24 16:40:35]: Step 16 of 20 Backing up system data

[IC-F7811 2012-09-24 16:40:36]: Step 16 of 20 Backing up system data

[IC-F7712 2012-09-24 16:40:37]: Step 17 of 20 Setting up shared partition data

[IC-F7811 2012-09-24 16:40:37]: Step 17 of 20 Setting up shared partition data

[IC-F7712 2012-09-24 16:40:37]: Step 18 of 20 Checking package sanity in installation

[IC-F7712 2012-09-24 16:40:37]: Step 19 of 20 Unmounting and cleaning up temporary file systems

[IC-F7811 2012-09-24 16:40:37]: Step 18 of 20 Checking package sanity in installation

[IC-F7811 2012-09-24 16:40:37]: Step 19 of 20 Unmounting and cleaning up temporary file systems

[IC-F7712 2012-09-24 16:40:40]: Step 20 of 20 Setting da0s1 as new active partition

[IC-F7811 2012-09-24 16:40:41]: Step 20 of 20 Setting da0s1 as new active partition

[IC-F7712 2012-09-24 16:40:50]: ——- re0: ——-

[IC-F7811 2012-09-24 16:40:52]: ——- re0: ——-

[IC-F7712 2012-09-24 16:41:36]: Step 1 of 20 Creating temporary file system

[IC-F7712 2012-09-24 16:41:36]: Step 2 of 20 Determining installation source

[IC-F7712 2012-09-24 16:41:37]: Step 3 of 20 Processing format options

[IC-F7712 2012-09-24 16:41:37]: Step 4 of 20 Determining installation slice

[IC-F7712 2012-09-24 16:41:38]: Step 5 of 20 Creating and labeling new slices

[IC-F7712 2012-09-24 16:41:38]: Step 6 of 20 Create and mount new file system

[IC-F7811 2012-09-24 16:41:39]: Step 1 of 20 Creating temporary file system

[IC-F7811 2012-09-24 16:41:39]: Step 2 of 20 Determining installation source

[IC-F7811 2012-09-24 16:41:40]: Step 3 of 20 Processing format options

[IC-F7811 2012-09-24 16:41:40]: Step 4 of 20 Determining installation slice

[IC-F7811 2012-09-24 16:41:41]: Step 5 of 20 Creating and labeling new slices

[IC-F7811 2012-09-24 16:41:42]: Step 6 of 20 Create and mount new file system

[IC-F7712 2012-09-24 16:41:49]: Step 7 of 20 Getting OS bundles

[IC-F7712 2012-09-24 16:41:50]: Step 8 of 20 Updating recovery media

[IC-F7811 2012-09-24 16:41:51]: Step 7 of 20 Getting OS bundles

[IC-F7811 2012-09-24 16:41:51]: Step 8 of 20 Updating recovery media

[IC-F7712 2012-09-24 16:42:15]: Step 9 of 20 Extracting incoming image

[IC-F7811 2012-09-24 16:42:19]: Step 9 of 20 Extracting incoming image

[IC-F7712 2012-09-24 16:44:01]: Step 10 of 20 Unpacking OS packages

[IC-F7712 2012-09-24 16:44:04]: Step 11 of 20 Mounting jbase package

[IC-F7811 2012-09-24 16:44:05]: Step 10 of 20 Unpacking OS packages

[IC-F7811 2012-09-24 16:44:07]: Step 11 of 20 Mounting jbase package

[IC-F7712 2012-09-24 16:44:36]: Step 12 of 20 Creating base OS symbolic links

[IC-F7811 2012-09-24 16:44:40]: Step 12 of 20 Creating base OS symbolic links

[IC-F7712 2012-09-24 16:46:01]: Step 13 of 20 Creating fstab

[IC-F7712 2012-09-24 16:46:01]: Step 14 of 20 Creating new system files

[IC-F7712 2012-09-24 16:46:01]: Step 15 of 20 Adding jbundle package

[IC-F7811 2012-09-24 16:46:06]: Step 13 of 20 Creating fstab

[IC-F7811 2012-09-24 16:46:06]: Step 14 of 20 Creating new system files

[IC-F7811 2012-09-24 16:46:06]: Step 15 of 20 Adding jbundle package

[IC-F7712 2012-09-24 16:49:41]: Step 16 of 20 Backing up system data

[IC-F7811 2012-09-24 16:49:45]: Step 16 of 20 Backing up system data

[IC-F7811 2012-09-24 16:49:47]: Step 17 of 20 Setting up shared partition data

[IC-F7811 2012-09-24 16:49:48]: Step 18 of 20 Checking package sanity in installation

[IC-F7811 2012-09-24 16:49:48]: Step 19 of 20 Unmounting and cleaning up temporary file systems

[IC-F7811 2012-09-24 16:49:51]: Step 20 of 20 Setting da0s1 as new active partition

[IC-F7712 2012-09-24 16:51:13]: Step 17 of 20 Setting up shared partition data

[IC-F7712 2012-09-24 16:51:14]: Step 18 of 20 Checking package sanity in installation

[IC-F7712 2012-09-24 16:51:14]: Step 19 of 20 Unmounting and cleaning up temporary file systems

[IC-F7712 2012-09-24 16:51:17]: Step 20 of 20 Setting da0s1 as new active partition

[Status   2012-09-24 16:51:32]: Load completed with 0 errors

[Status   2012-09-24 16:51:32]: Reboot is required to complete upgrade

[Status   2012-09-24 16:51:32]: Rebooting FC-1

[FC-1     2012-09-24 16:51:33]: Waiting for FC-1 to terminate

[FC-1     2012-09-24 16:52:18]: Waiting for FC-1 to come back

[FC-1     2012-09-24 16:55:10]: Running Uptime Test for FC-1

[FC-1     2012-09-24 16:55:26]: Uptime Test for FC-1 Passed

[Status   2012-09-24 16:55:27]: FC-1 Booted successfully

[Status   2012-09-24 16:55:27]: Rebooting FC-0

[FC-0     2012-09-24 16:55:27]: Waiting for FC-0 to terminate

[FC-0     2012-09-24 16:56:12]: Waiting for FC-0 to come back

[FC-0     2012-09-24 16:59:06]: Running Uptime Test for FC-0

[FC-0     2012-09-24 16:59:22]: Uptime Test for FC-0 Passed

[Status   2012-09-24 16:59:22]: FC-0 Booted successfully

[Status   2012-09-24 16:59:22]: Rebooting IC-F7811

[IC-F7811 2012-09-24 16:59:28]: Waiting for IC-F7811 to terminate

[IC-F7811 2012-09-24 16:59:59]: Waiting for IC-F7811 to come back

[IC-F7811 2012-09-24 17:06:45]: Running Uptime Test for IC-F7811

[IC-F7811 2012-09-24 17:07:34]: Waiting for FM to be ready

[IC-F7811 2012-09-24 17:13:09]: Performing post-boot Health-Check

[IC-F7811 2012-09-24 17:14:24]: Waiting for routes to sync

[IC-F7811 2012-09-24 17:14:32]: Uptime Test for IC-F7811 Passed

[Status   2012-09-24 17:14:32]: IC-F7811 Booted successfully

[Status   2012-09-24 17:14:32]: Rebooting IC-F7712

[IC-F7712 2012-09-24 17:14:34]: Waiting for IC-F7712 to terminate

[IC-F7712 2012-09-24 17:15:07]: Waiting for IC-F7712 to come back

[IC-F7712 2012-09-24 17:22:03]: Running Uptime Test for IC-F7712

[IC-F7712 2012-09-24 17:22:47]: Waiting for FM to be ready

[IC-F7712 2012-09-24 17:29:28]: Performing post-boot Health-Check

[IC-F7712 2012-09-24 17:30:43]: Waiting for routes to sync

[IC-F7712 2012-09-24 17:30:49]: Uptime Test for IC-F7712 Passed

[Status   2012-09-24 17:30:50]: IC-F7712 Booted successfully

Success

Node Group Upgrades

The NWNG took around an hour (for 4 nodes) and around 40 minutes for a RSNG. This process upgrades a node at a time in the group and updates both slices. Currently there is no command to verify each slice’s version, it is a known issue.

Node Groups tested were 1 Network node group and 2 RSNGs:

  • NW-NG-0
  • RSNG01
  • RSNG02
  1. From the DG CLI initiate the config:
  • request system software nonstop-upgrade node-group GROUP-NAME FILE.rpm

Output:

root@FSASYDBRDQFAB01> …0-D20.4.rpm node-group NW-NG-0Upgrading target(s): NW-NG-0[NW-NG-0  2012-09-24 17:33:25]: Starting with package ftp://169.254.0.3/pub/images/12.2X50-D20.4/jinstall-qfx.tgz[NW-NG-0  2012-09-24 17:33:25]: Retrieving package[NW-NG-0  2012-09-24 17:34:47]: Pushing bundle to P6172-C[NW-NG-0  2012-09-24 17:35:20]: Pushing bundle to P6136-C[NW-NG-0  2012-09-24 17:35:53]: Pushing bundle to fpc4

[NW-NG-0  2012-09-24 17:36:27]: Pushing bundle to fpc5

[NW-NG-0  2012-09-24 17:36:59]: P6172-C: Validate package…

[NW-NG-0  2012-09-24 17:43:31]: P6136-C: Validate package…

[NW-NG-0  2012-09-24 17:43:31]: fpc4: Validate package…

[NW-NG-0  2012-09-24 17:43:41]: fpc5: Validate package…

[NW-NG-0  2012-09-24 17:43:41]: ——- P6172-C ——-

[NW-NG-0  2012-09-24 17:44:17]: Step 1 of 20 Creating temporary file system

[NW-NG-0  2012-09-24 17:44:17]: Step 2 of 20 Determining installation source

[NW-NG-0  2012-09-24 17:44:18]: Step 3 of 20 Processing format options

[NW-NG-0  2012-09-24 17:44:18]: Step 4 of 20 Determining installation slice

[NW-NG-0  2012-09-24 17:44:18]: Step 5 of 20 Creating and labeling new slices

[NW-NG-0  2012-09-24 17:44:19]: Step 6 of 20 Create and mount new file system

[NW-NG-0  2012-09-24 17:44:27]: Step 7 of 20 Getting OS bundles

[NW-NG-0  2012-09-24 17:44:27]: Step 8 of 20 Updating recovery media

[NW-NG-0  2012-09-24 17:44:48]: Step 9 of 20 Extracting incoming image

[NW-NG-0  2012-09-24 17:46:02]: Step 10 of 20 Unpacking OS packages

[NW-NG-0  2012-09-24 17:46:07]: Step 11 of 20 Mounting jbase package

[NW-NG-0  2012-09-24 17:46:33]: Step 12 of 20 Creating base OS symbolic links

[NW-NG-0  2012-09-24 17:47:33]: Step 13 of 20 Creating fstab

[NW-NG-0  2012-09-24 17:47:33]: Step 14 of 20 Creating new system files

[NW-NG-0  2012-09-24 17:47:34]: Step 15 of 20 Adding jbundle package

[NW-NG-0  2012-09-24 17:50:07]: Step 16 of 20 Backing up system data

[NW-NG-0  2012-09-24 17:50:08]: Step 17 of 20 Setting up shared partition data

[NW-NG-0  2012-09-24 17:50:09]: Step 18 of 20 Checking package sanity in installation

[NW-NG-0  2012-09-24 17:50:09]: Step 19 of 20 Unmounting and cleaning up temporary file systems

[NW-NG-0  2012-09-24 17:50:12]: Step 20 of 20 Setting da0s2 as new active partition

[NW-NG-0  2012-09-24 17:50:23]: ——- P6136-C ——-

[NW-NG-0  2012-09-24 17:50:23]: Step 1 of 20 Creating temporary file system

[NW-NG-0  2012-09-24 17:50:23]: Step 2 of 20 Determining installation source

[NW-NG-0  2012-09-24 17:50:23]: Step 3 of 20 Processing format options

[NW-NG-0  2012-09-24 17:50:23]: Step 4 of 20 Determining installation slice

[NW-NG-0  2012-09-24 17:50:23]: Step 5 of 20 Creating and labeling new slices

[NW-NG-0  2012-09-24 17:50:23]: Step 6 of 20 Create and mount new file system

[NW-NG-0  2012-09-24 17:50:23]: Step 7 of 20 Getting OS bundles

[NW-NG-0  2012-09-24 17:50:23]: Step 8 of 20 Updating recovery media

[NW-NG-0  2012-09-24 17:50:23]: Step 9 of 20 Extracting incoming image

[NW-NG-0  2012-09-24 17:50:23]: Step 10 of 20 Unpacking OS packages

[NW-NG-0  2012-09-24 17:50:23]: Step 11 of 20 Mounting jbase package

[NW-NG-0  2012-09-24 17:50:23]: Step 12 of 20 Creating base OS symbolic links

[NW-NG-0  2012-09-24 17:50:23]: Step 13 of 20 Creating fstab

[NW-NG-0  2012-09-24 17:50:23]: Step 14 of 20 Creating new system files

[NW-NG-0  2012-09-24 17:50:23]: Step 15 of 20 Adding jbundle package

[NW-NG-0  2012-09-24 17:50:23]: Step 16 of 20 Backing up system data

[NW-NG-0  2012-09-24 17:50:23]: Step 17 of 20 Setting up shared partition data

[NW-NG-0  2012-09-24 17:50:23]: Step 18 of 20 Checking package sanity in installation

[NW-NG-0  2012-09-24 17:50:23]: Step 19 of 20 Unmounting and cleaning up temporary file systems

[NW-NG-0  2012-09-24 17:50:23]: Step 20 of 20 Setting da0s2 as new active partition

[NW-NG-0  2012-09-24 17:50:27]: Step 1 of 20 Creating temporary file system

[NW-NG-0  2012-09-24 17:50:27]: Step 2 of 20 Determining installation source

[NW-NG-0  2012-09-24 17:50:27]: Step 3 of 20 Processing format options

[NW-NG-0  2012-09-24 17:50:27]: Step 4 of 20 Determining installation slice

[NW-NG-0  2012-09-24 17:50:27]: Step 5 of 20 Creating and labeling new slices

[NW-NG-0  2012-09-24 17:50:27]: Step 6 of 20 Create and mount new file system

[NW-NG-0  2012-09-24 17:50:27]: Step 7 of 20 Getting OS bundles

[NW-NG-0  2012-09-24 17:50:27]: Step 8 of 20 Updating recovery media

[NW-NG-0  2012-09-24 17:50:27]: Step 9 of 20 Extracting incoming image

[NW-NG-0  2012-09-24 17:50:27]: Step 10 of 20 Unpacking OS packages

[NW-NG-0  2012-09-24 17:50:27]: Step 11 of 20 Mounting jbase package

[NW-NG-0  2012-09-24 17:50:27]: Step 12 of 20 Creating base OS symbolic links

[NW-NG-0  2012-09-24 17:50:27]: Step 13 of 20 Creating fstab

[NW-NG-0  2012-09-24 17:50:27]: Step 14 of 20 Creating new system files

[NW-NG-0  2012-09-24 17:50:27]: Step 15 of 20 Adding jbundle package

[NW-NG-0  2012-09-24 17:50:27]: Step 16 of 20 Backing up system data

[NW-NG-0  2012-09-24 17:50:27]: Step 17 of 20 Setting up shared partition data

[NW-NG-0  2012-09-24 17:50:27]: Step 18 of 20 Checking package sanity in installation

[NW-NG-0  2012-09-24 17:50:27]: Step 19 of 20 Unmounting and cleaning up temporary file systems

[NW-NG-0  2012-09-24 17:50:27]: Step 20 of 20 Setting da0s2 as new active partition

[NW-NG-0  2012-09-24 17:50:27]: Step 1 of 20 Creating temporary file system

[NW-NG-0  2012-09-24 17:50:27]: Step 2 of 20 Determining installation source

[NW-NG-0  2012-09-24 17:50:27]: Step 3 of 20 Processing format options

[NW-NG-0  2012-09-24 17:50:27]: Step 4 of 20 Determining installation slice

[NW-NG-0  2012-09-24 17:50:27]: Step 5 of 20 Creating and labeling new slices

[NW-NG-0  2012-09-24 17:50:27]: Step 6 of 20 Create and mount new file system

[NW-NG-0  2012-09-24 17:50:27]: Step 7 of 20 Getting OS bundles

[NW-NG-0  2012-09-24 17:50:27]: Step 8 of 20 Updating recovery media

[NW-NG-0  2012-09-24 17:50:27]: Step 9 of 20 Extracting incoming image

[NW-NG-0  2012-09-24 17:50:27]: Step 10 of 20 Unpacking OS packages

[NW-NG-0  2012-09-24 17:50:27]: Step 11 of 20 Mounting jbase package

[NW-NG-0  2012-09-24 17:50:27]: Step 12 of 20 Creating base OS symbolic links

[NW-NG-0  2012-09-24 17:50:27]: Step 13 of 20 Creating fstab

[NW-NG-0  2012-09-24 17:50:27]: Step 14 of 20 Creating new system files

[NW-NG-0  2012-09-24 17:50:27]: Step 15 of 20 Adding jbundle package

[NW-NG-0  2012-09-24 17:50:27]: Step 16 of 20 Backing up system data

[NW-NG-0  2012-09-24 17:50:27]: Step 17 of 20 Setting up shared partition data

[NW-NG-0  2012-09-24 17:50:27]: Step 18 of 20 Checking package sanity in installation

[NW-NG-0  2012-09-24 17:50:27]: Step 19 of 20 Unmounting and cleaning up temporary file systems

[NW-NG-0  2012-09-24 17:50:27]: Step 20 of 20 Setting da0s2 as new active partition

[NW-NG-0  2012-09-24 17:50:27]: Starting with package ftp://169.254.0.3/pub/images/12.2X50-D20.4/jinstall-dc-re.tgz

[NW-NG-0  2012-09-24 17:50:27]: Retrieving package

[NW-NG-0  2012-09-24 17:51:35]: Pushing bundle to re0

[NW-NG-0  2012-09-24 17:52:09]: re0: Validate package…

[NW-NG-0  2012-09-24 17:53:56]: re1: Validate package…

[NW-NG-0  2012-09-24 17:55:53]: Rebooting Backup RE

[NW-NG-0  2012-09-24 17:59:56]: Initiating Chassis In-Service-Upgrade

[NW-NG-0  2012-09-24 18:00:16]: Upgrading group: 2 fpc: 2

[NW-NG-0  2012-09-24 18:10:08]: Upgrade complete for group:2

[NW-NG-0  2012-09-24 18:10:08]: Upgrading group: 3 fpc: 3

[NW-NG-0  2012-09-24 18:19:58]: Upgrade complete for group:3

[NW-NG-0  2012-09-24 18:19:58]: Upgrading group: 4 fpc: 4

[NW-NG-0  2012-09-24 18:29:45]: Upgrade complete for group:4

[NW-NG-0  2012-09-24 18:29:45]: Upgrading group: 5 fpc: 5

[NW-NG-0  2012-09-24 18:39:32]: Upgrade complete for group:5

[NW-NG-0  2012-09-24 18:39:32]: Finished processing all upgrade groups, last group :5

[NW-NG-0  2012-09-24 18:39:37]: Preparing for Switchover

[NW-NG-0  2012-09-24 18:39:54]: Switchover Completed

[Status   2012-09-24 18:39:54]: Upgrade completed with 0 errors

Success

root@FSASYDBRDQFAB01> …0-D20.4.rpm node-group RSNG01

Upgrading target(s): RSNG01

[RSNG01   2012-09-25 11:44:47]: Starting with package ftp://169.254.0.3/pub/images/12.2X50-D20.4/jinstall-qfx.tgz

[RSNG01   2012-09-25 11:44:47]: Retrieving package

[RSNG01   2012-09-25 11:46:55]: Pushing bundle to P6167-C

[RSNG01   2012-09-25 11:47:27]: P6167-C: Validate package…

[RSNG01   2012-09-25 11:53:38]: P6185-C: Validate package…

[RSNG01   2012-09-25 11:54:16]: ——- P6167-C ——-

[RSNG01   2012-09-25 11:54:53]: Step 1 of 20 Creating temporary file system

[RSNG01   2012-09-25 11:54:53]: Step 2 of 20 Determining installation source

[RSNG01   2012-09-25 11:54:54]: Step 3 of 20 Processing format options

[RSNG01   2012-09-25 11:54:54]: Step 4 of 20 Determining installation slice

[RSNG01   2012-09-25 11:54:55]: Step 5 of 20 Creating and labeling new slices

[RSNG01   2012-09-25 11:54:55]: Step 6 of 20 Create and mount new file system

[RSNG01   2012-09-25 11:55:03]: Step 7 of 20 Getting OS bundles

[RSNG01   2012-09-25 11:55:03]: Step 8 of 20 Updating recovery media

[RSNG01   2012-09-25 11:55:25]: Step 9 of 20 Extracting incoming image

[RSNG01   2012-09-25 11:56:40]: Step 10 of 20 Unpacking OS packages

[RSNG01   2012-09-25 11:56:45]: Step 11 of 20 Mounting jbase package

[RSNG01   2012-09-25 11:57:09]: Step 12 of 20 Creating base OS symbolic links

[RSNG01   2012-09-25 11:58:10]: Step 13 of 20 Creating fstab

[RSNG01   2012-09-25 11:58:11]: Step 14 of 20 Creating new system files

[RSNG01   2012-09-25 11:58:11]: Step 15 of 20 Adding jbundle package

[RSNG01   2012-09-25 12:00:48]: Step 16 of 20 Backing up system data

[RSNG01   2012-09-25 12:00:50]: Step 17 of 20 Setting up shared partition data

[RSNG01   2012-09-25 12:00:50]: Step 18 of 20 Checking package sanity in installation

[RSNG01   2012-09-25 12:00:50]: Step 19 of 20 Unmounting and cleaning up temporary file systems

[RSNG01   2012-09-25 12:00:54]: Step 20 of 20 Setting da0s2 as new active partition

[RSNG01   2012-09-25 12:01:05]: ——- P6185-C – master ——-

[RSNG01   2012-09-25 12:01:05]: Step 1 of 20 Creating temporary file system

[RSNG01   2012-09-25 12:01:05]: Step 2 of 20 Determining installation source

[RSNG01   2012-09-25 12:01:05]: Step 3 of 20 Processing format options

[RSNG01   2012-09-25 12:01:05]: Step 4 of 20 Determining installation slice

[RSNG01   2012-09-25 12:01:05]: Step 5 of 20 Creating and labeling new slices

[RSNG01   2012-09-25 12:01:05]: Step 6 of 20 Create and mount new file system

[RSNG01   2012-09-25 12:01:05]: Step 7 of 20 Getting OS bundles

[RSNG01   2012-09-25 12:01:05]: Step 8 of 20 Updating recovery media

[RSNG01   2012-09-25 12:01:05]: Step 9 of 20 Extracting incoming image

[RSNG01   2012-09-25 12:01:05]: Step 10 of 20 Unpacking OS packages

[RSNG01   2012-09-25 12:01:05]: Step 11 of 20 Mounting jbase package

[RSNG01   2012-09-25 12:01:05]: Step 12 of 20 Creating base OS symbolic links

[RSNG01   2012-09-25 12:01:05]: Step 13 of 20 Creating fstab

[RSNG01   2012-09-25 12:01:05]: Step 14 of 20 Creating new system files

[RSNG01   2012-09-25 12:01:05]: Step 15 of 20 Adding jbundle package

[RSNG01   2012-09-25 12:01:05]: Step 16 of 20 Backing up system data

[RSNG01   2012-09-25 12:01:05]: Step 17 of 20 Setting up shared partition data

[RSNG01   2012-09-25 12:01:05]: Step 18 of 20 Checking package sanity in installation

[RSNG01   2012-09-25 12:01:05]: Step 19 of 20 Unmounting and cleaning up temporary file systems

[RSNG01   2012-09-25 12:01:05]: Step 20 of 20 Setting da0s2 as new active partition

[RSNG01   2012-09-25 12:01:51]: Rebooting Backup RE

[RSNG01   2012-09-25 12:01:51]: ——- Rebooting P6167-C ——-

[RSNG01   2012-09-25 12:08:49]: Initiating Chassis In-Service-Upgrade

[RSNG01   2012-09-25 12:09:09]: Upgrading group: 0 fpc: 0

[RSNG01   2012-09-25 12:11:15]: Upgrade complete for group:0

[RSNG01   2012-09-25 12:11:15]: Upgrading group: 1 fpc: 1

[RSNG01   2012-09-25 12:13:20]: Upgrade complete for group:1

[RSNG01   2012-09-25 12:13:20]: Finished processing all upgrade groups, last group :1

[RSNG01   2012-09-25 12:13:24]: Preparing for Switchover

[RSNG01   2012-09-25 12:14:15]: Switchover Completed

[Status   2012-09-25 12:14:15]: Upgrade completed with 0 errors

Success

Conclusion

The NSSU QFabric upgrade is a very simple and well polished process. Apart from being very time consuming, it’s great and I really like how it’s been designed and implemented. It’s quite verbose and keeps the operator well informed, which I like, loving knowing what is actually going on. I also like (some may argue this is bad) the automatic upgrade of each SCB on the Interconnects and each slice on the nodes, saving that extra step post upgrade, but does make rollback harder.

Well done Juniper, this is another great part of the QFabric Solution!

P.s. Just give me a ssh client and automatic system archival.

Juniper SRX Chassis Cluster RG0 Nagios Check

I was required to check (as this customer did not have a trap collector) which node was active for redundancy group 0 on a SRX cluster. So I thought I would check for a SNMP OID that is only presented by the active RG0 node. This script uses snmpwalk and is configured to use SNMP v2c (this can be easily changed). It has been tested on:

  • CentOS 5
  • Junos 11.4R2
  • SNMP v2c

Here is the little hacky shell script:

[bash]
#!/bin/bash

# Cooper Lees <me@cooperlees.com>
# Dirty Cluster RG0 checker
# Lasted Updated: 20120818

HOST=$1
COMMUNITY=$2

if [ "$HOST" == "" ] || [ "$COMMUNITY" == "" ]; then
echo "ERROR: No host or SNMP community specified"
exit 2
fi

SNMPOUTPUT=$(snmpwalk -v 2c -c $COMMUNITY $HOST 1.3.6.1.4.1.2636.3.1.14.1.7)

echo $SNMPOUTPUT | grep "INTEGER: 2" > /dev/null
if [ $? == 0 ]; then
echo "Host $HOST is the Chassis cluster ACTIVE RE"
exit 0
fi

echo $SNMPOUTPUT | grep "No Such Object available on this agent at this OID" > /dev/null
if [ $? == 0 ]; then
echo "Host $HOST is the INACTIVE RE"
exit 2
fi

echo "WTF – Something is not right …"
exit 3
[/bash]

It checks for the “jnxRedundancyState” OID – this OID reports on RE states and is only accurate on Junos routers (e.g. M and MX series etc.).

Enjoy …

SRX Branch Chassis Cluster Ports

Here is a table of the ports that are used for chassis cluster control link and management ports on Branch SRX devices.

The quoted ports are the ‘stand alone’ non clustered port names (not node1’s port names once clustered). In a SRX cluster the PIM slots on node1 start at the last PIM slot of node0 + 1. For example, a SRX240 cluster’s node1 starts at PIM 5. It’s control link port is effectively ge-5/0/1).

Model FXP0 (Management) FXP1 (Control Link)
SRX100 fe-0/0/6 fe-0/0/7
SRX210 fe-0/0/6 fe-0/0/7
SRX220 ge-0/0/6 (> 11.0) ge-0/0/7
SRX240 ge-0/0/0 ge-0/0/1
SRX550 ge-0/0/0 ge-0/0/1
SRX650 ge-0/0/0 ge-0/0/1

 *fab0 and fab1 interfaces (Data Link) are always configurable, e.g.:

  • set interfaces fab0 fabric-options member-interfaces ge-0/0/2
  • set interfaces fab1 fabric-options member-interfaces ge-5/0/2