rssLink RSS for all categories
 
icon_red
icon_green
icon_red
icon_red
icon_red
icon_green
icon_green
icon_red
icon_red
icon_red
icon_orange
icon_green
icon_green
icon_green
icon_red
icon_blue
icon_red
icon_orange
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_orange
icon_green
 

FS#2272 — FS#6241 — Upgrades ASR 9K Roubaix

Attached to Project— Network
Maintenance
the whole network
CLOSED
100%
We are going to upgrade the 2 routers "core" here in Roubaix in order to add capacity.

These chassis are now full in terms of linecard 8x10G. . We are going to switch to the new RSP 440 which will allow us to replace some of the line cards for the new 24x10G. The procedure is as follows :

- upgrade software
- replacement of the RSPs (stopping/ complete rerestarting of the necessary chassis)
-replacement of 2 of the linecards 8G by 24g


These intervention will take place during the night. We will start this night (from 10 to 11 ) with rbx-g1-a9 at 00:00 CET. No impact on the trafic is anticipated because we are going to force the trafic by rbx-g2 during the workings.
Date:  Tuesday, 31 January 2012, 14:44PM
Reason for closing:  Done
Comment by OVH - Thursday, 12 January 2012, 21:11PM

We begin the workings. We are going to reroute a part of the trafic of the transits at London and Amsterdam to Paris in order to divert the links of rbx-g1 to ldn-1/5 and Amsterdam 1/5.


Comment by OVH - Thursday, 12 January 2012, 21:11PM

We are going to prepare the router to the set up
of the new version of IOS XR.


Comment by OVH - Thursday, 12 January 2012, 21:12PM

RP/0/RSP1/CPU0:rbx-g1-a9(admin)#show install inactive
Tue Jan 10 23:55:33.960 UTC
Secure Domain Router: Owner

There are no inactive packages on the SDR.

RP/0/RSP1/CPU0:rbx-g1-a9#sh install summary
Tue Jan 10 23:56:25.147 UTC
Active Packages:
disk0:asr9k-mini-p-4.0.1
disk0:asr9k-doc-p-4.0.1
disk0:asr9k-k9sec-p-4.0.1
disk0:asr9k-mpls-p-4.0.1
disk0:asr9k-mgbl-p-4.0.1
disk0:asr9k-mcast-p-4.0.1


Comment by OVH - Thursday, 12 January 2012, 21:12PM

(admin) install add source disk0:// asr9k-doc-p.pie-4.2.0 asr9k-k9sec-p.pie-4.2.0 asr9k-mcast-p.pie-4.2.0 asr9k-mgbl-p.pie-4.2.0 asr9k-mini-p.pie-4.2.0 asr9k-mpls-p.pie-4.2.0 asr9k-upgrade-p.pie-4.2.0 synchronous


Comment by OVH - Thursday, 12 January 2012, 21:12PM

Meanwhile on g2:

RP/0/RSP0/CPU0:rbx-g2-a9#delete disk0:ASR9K-iosxr-k9-4.1.1.tar

RP/0/RSP0/CPU0:rbx-g2-a9#sh install inactive
Wed Jan 11 00:19:21.016 UTC
Node 0/RSP0/CPU0 [RP] [SDR: Owner]
Boot Device: disk0:
Inactive Packages:
disk0:asr9k-upgrade-p-4.1.0
disk0:asr9k-mini-p-4.1.0
disk0:asr9k-doc-p-4.1.0
disk0:asr9k-k9sec-p-4.1.0
disk0:asr9k-mpls-p-4.1.0
disk0:asr9k-mgbl-p-4.1.0
disk0:asr9k-mcast-p-4.1.0
[...]
RP/0/RSP0/CPU0:rbx-g2-a9#admin
Wed Jan 11 00:19:38.546 UTC
RP/0/RSP0/CPU0:rbx-g2-a9(admin)#install remove inactive
Wed Jan 11 00:19:44.906 UTC
Install operation 26 '(admin) install remove inactive' started by user 'octave' via CLI at 00:19:45 UTC Wed Jan 11 2012.
Info: This operation will remove the following packages:
Info: disk0:asr9k-upgrade-p-4.1.0
Info: disk0:asr9k-doc-4.1.0
Info: disk0:asr9k-adv-video-4.1.0
Info: disk0:asr9k-sbc-4.1.0
Info: disk0:asr9k-k9sec-4.1.0
[...]
Proceed with removing these packages? [confirm]
The install operation will continue asynchronously.


Comment by OVH - Thursday, 12 January 2012, 21:13PM

We changed the cost OSPF of g1 to make it out
of the routing.


Comment by OVH - Thursday, 12 January 2012, 21:14PM

Let's go !

RP/0/RSP1/CPU0:rbx-g1-a9(admin)#install activate ?
disk0:asr9k-doc-p-4.2.0 Package to activate
disk0:asr9k-k9sec-p-4.2.0 Package to activate
disk0:asr9k-mcast-p-4.2.0 Package to activate
disk0:asr9k-mgbl-p-4.2.0 Package to activate
disk0:asr9k-mini-p-4.2.0 Package to activate
disk0:asr9k-mpls-p-4.2.0 Package to activate
disk0:asr9k-upgrade-p-4.2.0 Package to activate
id The id of the install add operation containing packages to activate

RP/0/RSP1/CPU0:rbx-g1-a9(admin)#install activate disk0:*4.2.0* synchronous
[...]
Proceed with this install operation (y/n)? [y]


Comment by OVH - Thursday, 12 January 2012, 21:14PM

Proceed with this install operation (y/n)? [y]
Info: Install Method: Parallel Reload
Info: The changes made to software configurations will not be persistent across system reloads. Use the command '(admin) install commit' to make changes persistent.
Info: Please verify that the system is consistent following the software change using the following commands:
Info: show system verify
Info: install verify packages
Install operation 132 completed successfully at 00:41:37 UTC Wed Jan 11 2012.
RP/0/RSP1/CPU0:rbx-g1-a9(admin)#
RP/0/RSP1/CPU0:rbx-g1-a9(admin)#
RP/0/RSP1/CPU0:rbx-g1-a9(admin)#
RP/0/RSP1/CPU0:rbx-g1-a9(admin)#
RP/0/RSP1/CPU0:rbx-g1-a9(admin)#

It's done .... it's rebooted.


Comment by OVH - Thursday, 12 January 2012, 21:15PM

This (D)RP Node is not ready or active for login /configuration

This (D)RP Node is not ready or active for login /configuration

This (D)RP Node is not ready or active for login /configuration

*** insthelper node 0/RSP0/CPU0: rebooting in 35 seconds due to reload upgrade 132 ...
[0x79701f673bab5] Record Reboot History, reboot cause = 0x4000014, descr = Cause: Install operation. Shutdown standby node gracefully. Process: redcon
Traceback: 4c223c30 4rebooting

Selecting ROMMON Image... B
DDR in Interleaved mode
POST 1 : PASSED : code 0 : DDR2 Memory Quick Test

CPU Reset Reason = 0x000d
POST 2 : PASSED : code 0 : FPGA Flash Image CRC Checks

Loading Field Programmable Devices:
FPGA 0-B PROGRAMMED : image: 0xff500028 - 0xff576cca, et: 117ms
FPGA 1-B PROGRAMMED : image: 0xff400028 - 0xff4d1034, et: 206ms
FPGA 2-B PROGRAMMED : image: 0xff100028 - 0xff276358, et: 369ms
FPGA 3-B PROGRAMMED : image: 0xff000028 - 0xff0454a8, et: 69ms

System Bootstrap, Version 1.05(20101118:025914) [ASR9K ROMMON],
Copyright (c) 1994-2010 by Cisco Systems, Inc.
Compiled Wed 17-Nov-10 18:59 by saurabja
[...]


Comment by OVH - Thursday, 12 January 2012, 21:15PM

The router is up.

RP/0/RSP0/CPU0:rbx-g1-a9(admin)#sh install summary
Wed Jan 11 00:53:30.008 UTC
Default Profile:
SDRs:
Owner
Active Packages:
disk0:asr9k-upgrade-p-4.2.0
disk0:asr9k-mini-p-4.2.0
disk0:asr9k-doc-p-4.2.0
disk0:asr9k-k9sec-p-4.2.0
disk0:asr9k-mpls-p-4.2.0
disk0:asr9k-mgbl-p-4.2.0
disk0:asr9k-mcast-p-4.2.0

RP/0/RSP0/CPU0:rbx-g1-a9(admin)#install deactivate disk0:asr9k-upgrade-p-4.2.0 sync

RP/0/RSP0/CPU0:rbx-g1-a9(admin)#install remove disk0:asr9k-upgrade-p-4.2.0 sync
Wed Jan 11 00:56:34.030 UTC
Install operation 134 '(admin) install remove disk0:asr9k-upgrade-p-4.2.0 synchronous' started by user 'octave' via CLI at 00:56:34 UTC Wed Jan 11 2012.
Info: This operation will remove the following package:
Info: disk0:asr9k-upgrade-p-4.2.0
Info: After this install remove the following install rollback point will no longer be reachable, as the required packages will not be present:
Info: 132
Proceed with removing these packages? [confirm]
Install operation 134 completed successfully at 00:56:52 UTC Wed Jan 11 2012.


Comment by OVH - Thursday, 12 January 2012, 21:15PM

RP/0/RSP0/CPU0:rbx-g1-a9(admin)#sh install summary
Wed Jan 11 00:58:56.412 UTC
Default Profile:
SDRs:
Owner
Active Packages:
disk0:asr9k-mini-p-4.2.0
disk0:asr9k-doc-p-4.2.0
disk0:asr9k-k9sec-p-4.2.0
disk0:asr9k-mpls-p-4.2.0
disk0:asr9k-mgbl-p-4.2.0
disk0:asr9k-mcast-p-4.2.0


Comment by OVH - Thursday, 12 January 2012, 21:16PM

RP/0/RSP0/CPU0:rbx-g1-a9(admin)#sh install inactive
Wed Jan 11 00:59:08.043 UTC
Secure Domain Router: Owner

Node 0/RSP0/CPU0 [RP] [SDR: Owner]
Boot Device: disk0:
Inactive Packages:
disk0:asr9k-mini-p-4.0.1
disk0:asr9k-doc-p-4.0.1
disk0:asr9k-k9sec-p-4.0.1
disk0:asr9k-mpls-p-4.0.1
disk0:asr9k-mgbl-p-4.0.1
disk0:asr9k-mcast-p-4.0.1
[...]

RP/0/RSP0/CPU0:rbx-g1-a9(admin)#install remove ?
WORD <device><directory> (may include '*' or '?')
disk0:asr9K-doc-supp-4.0.1 Package to remove
disk0:asr9k-base-4.0.1 Package to remove
disk0:asr9k-cpp-4.0.1 Package to remove
[...]

RP/0/RSP0/CPU0:rbx-g1-a9(admin)#install remove inactive synchronous
Wed Jan 11 01:00:11.094 UTC
Install operation 136 '(admin) install remove inactive synchronous' started by user 'octave' via CLI at 01:00:11 UTC Wed Jan 11 2012.
Info: This operation will remove the following packages:
Info: disk0:asr9k-mpls-p-4.0.1
Info: disk0:asr9k-mini-p-4.0.1
Info: disk0:asr9k-mgbl-p-4.0.1
Info: disk0:asr9k-mcast-p-4.0.1
Info: disk0:asr9k-k9sec-p-4.0.1
Info: disk0:asr9k-doc-p-4.0.1
Info: After this install remove the following install rollback points will no longer be reachable, as the required packages will not be present:
Info: 114, 116
Proceed with removing these packages? [confirm]


Comment by OVH - Thursday, 12 January 2012, 21:16PM

RP/0/RSP0/CPU0:rbx-g1-a9(admin)#upgrade hw-module fpd all location all
Wed Jan 11 01:04:46.813 UTC

***** UPGRADE WARNING MESSAGE: *****
* This upgrade operation has a maximum timout of 90 minutes. *
* If you are executing the cmd for one specific location and *
* card in that location reloads or goes down for some reason *
* you can press CTRL-C to get back the RP's prompt. *
* If you are executing the cmd for _all_ locations and a node *
* reloads or is down please allow other nodes to finish the *
* upgrade process before pressing CTRL-C. *

% RELOAD REMINDER:
- The upgrade operation of the target module will not interrupt its normal
operation. However, for the changes to take effect, the target module
will need to be manually reloaded after the upgrade operation. This can
be accomplished with the use of "hw-module <target> reload" command.
- If automatic reload operation is desired after the upgrade, please use
the "reload" option at the end of the upgrade command.
- The output of "show hw-module fpd location" command will not display
correct version information after the upgrade if the target module is
not reloaded.
NOTE: Chassis CLI will not be accessible while upgrade is in progress.
Continue? [confirm]
This can take some time for a full chassis.
Ensure that system is not power cycled during the upgrades.
Please consult the documentation for more information.
Continue ? [no]: yes


FPD upgrade in progress on some hardware, reload/configuration change
on those is not recommended as it might cause HW programming failure
and result in RMA of the hardware.


Comment by OVH - Thursday, 12 January 2012, 21:16PM

Successfully upgraded fpga3 for A9K-RSP-4G on location 0/RSP0/CPU0 from 1.18 to 1.23
Successfully upgraded fpga3 for A9K-RSP-4G on location 0/RSP1/CPU0 from 1.18 to 1.23
FPD upgrade in progress. Max timeout remaining 89 min.
Successfully upgraded cpld1 for A9K-8T-L on location 0/4/CPU0 from 1.00 to 1.02
Successfully upgraded cpld4 for A9K-8T-L on location 0/4/CPU0 from 1.00 to 1.03
FPD upgrade in progress. Max timeout remaining 88 min.
FPD upgrade in progress. Max timeout remaining 87 min.
FPD upgrade in progress. Max timeout remaining 86 min.
FPD upgrade in progress. Max timeout remaining 85 min.
FPD upgrade in progress. Max timeout remaining 84 min.
Successfully upgraded fpga for A9K-8T-L on location 0/6/CPU0 from 1.00 to 1.02
Successfully upgraded cpld1 for A9K-8T-L on location 0/6/CPU0 from 1.00 to 1.02
Successfully upgraded cpld4 for A9K-8T-L on location 0/6/CPU0 from 1.00 to 1.03
Successfully upgraded fpga for A9K-8T-L on location 0/6/CPU0 from 1.00 to 1.02
FPD upgrade in progress. Max timeout remaining 83 min.
FPD upgrade in progress. Max timeout remaining 82 min.
Successfully upgraded fpga for A9K-8T-L on location 0/2/CPU0 from 1.00 to 1.02
Successfully upgraded cpld1 for A9K-8T-L on location 0/2/CPU0 from 1.00 to 1.02
Successfully upgraded cpld4 for A9K-8T-L on location 0/2/CPU0 from 1.00 to 1.03
Successfully upgraded fpga for A9K-8T-L on location 0/2/CPU0 from 1.00 to 1.02
Successfully upgraded fpga for A9K-8T-L on location 0/1/CPU0 from 1.00 to 1.02
Successfully upgraded cpld1 for A9K-8T-L on location 0/1/CPU0 from 1.00 to 1.02
Successfully upgraded cpld4 for A9K-8T-L on location 0/1/CPU0 from 1.00 to 1.03
Successfully upgraded fpga for A9K-8T-L on location 0/1/CPU0 from 1.00 to 1.02
FPD upgrade in progress. Max timeout remaining 81 min.
Successfully upgraded fpga for A9K-8T-L on location 0/5/CPU0 from 1.00 to 1.02
Successfully upgraded cpld1 for A9K-8T-L on location 0/5/CPU0 from 1.00 to 1.02
Successfully upgraded cpld4 for A9K-8T-L on location 0/5/CPU0 from 1.00 to 1.03
Successfully upgraded fpga for A9K-8T-L on location 0/5/CPU0 from 1.00 to 1.02
Successfully upgraded fpga for A9K-8T-L on location 0/0/CPU0 from 1.00 to 1.02
Successfully upgraded cpld1 for A9K-8T-L on location 0/0/CPU0 from 1.00 to 1.02
Successfully upgraded cpld4 for A9K-8T-L on location 0/0/CPU0 from 1.00 to 1.03
Successfully upgraded fpga for A9K-8T-L on location 0/0/CPU0 from 1.00 to 1.02
Successfully upgraded fpga for A9K-8T-L on location 0/7/CPU0 from 1.00 to 1.02
Successfully upgraded cpld1 for A9K-8T-L on location 0/7/CPU0 from 1.00 to 1.02
Successfully upgraded cpld4 for A9K-8T-L on location 0/7/CPU0 from 1.00 to 1.03
Successfully upgraded fpga for A9K-8T-L on location 0/7/CPU0 from 1.00 to 1.02
Successfully upgraded fpga for A9K-8T-L on location 0/3/CPU0 from 1.00 to 1.02
Successfully upgraded cpld1 for A9K-8T-L on location 0/3/CPU0 from 1.00 to 1.02
Successfully upgraded cpld4 for A9K-8T-L on location 0/3/CPU0 from 1.00 to 1.03
Successfully upgraded fpga for A9K-8T-L on location 0/3/CPU0 from 1.00 to 1.02


FPD upgrade has ended.


Comment by OVH - Thursday, 12 January 2012, 21:17PM

We are going no to take the RP2 out of the g1 chassis
and set up the new RP440.


Comment by OVH - Thursday, 12 January 2012, 21:19PM

It's done. The router is down again. We insert the new cards that we
are going to reconfigure from scratch since they are virgin.


Comment by OVH - Thursday, 12 January 2012, 21:20PM

Surprise ! ther's no image at all on the RP440.
They are not virgin !

##########################################################
System Bootstrap, Version 0.46 [ASR9K x86 ROMMON],
Copyright (c) 1994-2011 by Cisco Systems, Inc.
Compiled on Thu 09/22/2011 11:18:01.84 by myekkar

Rommon : 0.46
Ibex Peak : 6
Jasper Forest: 1.0
Zen-JF : 0.7.92
CBC : Part 1=16.114, Part 2=16.114, Act Part=2
Laguna : 0.9.0
Dao : 0.8.0
UTI : 4.9
Timex : 0.1.1
Board : 4
==========================================================
Sending Boot Image validation request.
HIT CTRL-C to abort
-----------
Interface link did not come up. Timed out.
Boot Image validation: Link failure
Sending Boot Image validation request.
HIT CTRL-C to abort
-----------
Interface link did not come up. Timed out.
Boot Image validation: Link failure
Sending Boot Image validation request.
HIT CTRL-C to abort
-----------
Interface link did not come up. Timed out.
Boot Image validation: Link failure
Sending Boot Image validation request.
HIT CTRL-C to abort
-----------
Interface link did not come up. Timed out.
Boot Image validation: Link failure
Sending Boot Image validation request.
HIT CTRL-C to abort


Comment by OVH - Thursday, 12 January 2012, 21:21PM

We put again the old RP2 and restart the g1.
We are going to put the trafic as before but in
4.2 and then update the g2.

After that, we are going to see with cisco how
to start these cards without image...


Comment by OVH - Thursday, 12 January 2012, 21:22PM

RP2 are in boot.

Selecting ROMMON Image... B
DDR in Interleaved mode
POST 1 : PASSED : code 0 : DDR2 Memory Quick Test

CPU Reset Reason = 0x0002
POST 2 : PASSED : code 0 : FPGA Flash Image CRC Checks

Loading Field Programmable Devices:
FPGA 0-B PROGRAMMED : image: 0xff500028 - 0xff576cca, et: 117ms
FPGA 1-B PROGRAMMED : image: 0xff400028 - 0xff4d1034, et: 206ms
FPGA 2-B PROGRAMMED : image: 0xff100028 - 0xff276358, et: 369ms
FPGA 3-B PROGRAMMED : image: 0xff000028 - 0xff0454a8, et: 69ms

System Bootstrap, Version 1.05(20101118:025914) [ASR9K ROMMON],
Copyright (c) 1994-2010 by Cisco Systems, Inc.
Compiled Wed 17-Nov-10 18:59 by saurabja

CPUCtrl: 1.17 [00000001/00000011]
ClkCtrl: 1.23 [00000001/00000017]
IntCtrl: 1.15 [00000001/0000000f]
Punt: 1.5 [00000001/00000005]
CBC: 1.2
BID: 0x0006


PPC 8641D (partnum 0x8004), Revision 03.00, (Core Version 02.02)
M8641 CLKIN: 66 Mhz
Core Clock: 1333 Mhz
MPX Clock: 533 Mhz
LBC Clock: 33 Mhz

POST 3 : PASSED : code 0 : Slot ID/Board Type Validity
PCI-E1: Ready as Root Complex
PCI-E2: Ready as Root Complex


set_chassis_type: chassis_type=0xef02fe found=TRUE
ASR9K (8641D PPC) platform with 4096 Mb of main memory

program load complete, entry point: 0x100000, size: 0x2ac18
program load complete, entry point: 0x100000, size: 0x2ac18
MBI Candidate = disk0:asr9k-os-mbi-4.2.0/0x100000/mbiasr9k-rp.vm

CARD_SLOT_NUMBER: 4
CPU_INSTANCE: 1
MBI Validation starts ...
Missing or illegal ip address for variable IP_ADDRESS

Mgt LAN 0 interface is selected
tsec_init_hw: configuring FE (port 2) for: Auto Speed, Auto Duplex

tsec_init_interface: hardware initialization completed
Interface link changed state to UP.
Interface link state up.

MBI validation sending request.
HIT CTRL-C to abort
.........
No MBI confirmation received from dSC

AUTOBOOT: Boot string = disk0:asr9k-os-mbi-4.2.0/0x100000/mbiasr9k-rp.vm,1;
AUTOBOOT: autobootstate=0, autobootcount=0, cmd=boot disk0:asr9k-os-mbi-4.2.0/0x100000/mbiasr9k-rp.vm
program load complete, entry point: 0x100000, size: 0x2ac18

MBI size from header = 21578140,Bootflash resident MBI filesize = 21578140
..............................


Comment by OVH - Thursday, 12 January 2012, 21:25PM

g1 is up. We're changing its costs

and after that we'll make a start on g2.

During the day we will see with cisco why the RP440
does not boot. According to the people from cisco
which we could join at this late hour, it's not normal.


Comment by OVH - Thursday, 12 January 2012, 21:26PM

The process ospv3 has a problem:

Jan 11 03:28:25 rbx-g2-a9.fr.eu 88274: RP/0/RSP0/CPU0:Jan 11 02:28:25 UTC: ospf[1009]: %ROUTING-OSPF-3-INTERNALERR : Internal error, path id out of range
Jan 11 03:28:25 rbx-g2-a9.fr.eu 88275: RP/0/RSP0/CPU0:Jan 11 02:28:25 UTC: ospf[1009]: %ROUTING-OSPF-3-INTERNALERR : Internal error, path id out of range
Jan 11 03:28:25 rbx-g2-a9.fr.eu 88276: RP/0/RSP0/CPU0:Jan 11 02:28:25 UTC: ospf[1009]: %ROUTING-OSPF-3-INTERNALERR : Internal error, path id out of range
Jan 11 03:28:25 rbx-g2-a9.fr.eu 88277: RP/0/RSP0/CPU0:Jan 11 02:28:25 UTC: ospf[1009]: %ROUTING-OSPF-3-INTERNALERR : Internal error, path id out of range
Jan 11 03:28:25 rbx-g2-a9.fr.eu 88278: RP/0/RSP0/CPU0:Jan 11 02:28:25 UTC: ospf[1009]: %ROUTING-OSPF-3-INTERNALERR : Internal error, path id out of range
Jan 11 03:28:25 rbx-g2-a9.fr.eu 88279: RP/0/RSP0/CPU0:Jan 11 02:28:25 UTC: ospf[1009]: %ROUTING-OSPF-3-INTERNALERR : Internal error, path id out of range
Jan 11 03:28:25 rbx-g2-a9.fr.eu 88280: RP/0/RSP0/CPU0:Jan 11 02:28:25 UTC: ospf[1009]: %ROUTING-OSPF-3-INTERNALERR : Internal error, path id out of range
Jan 11 03:28:25 rbx-g2-a9.fr.eu 88281: RP/0/RSP0/CPU0:Jan 11 02:28:25 UTC: ospf[1009]: %ROUTING-OSPF-3-INTERNALERR : Internal error, path id out of range
Jan 11 03:28:25 rbx-g2-a9.fr.eu 88282: RP/0/RSP0/CPU0:Jan 11 02:28:25 UTC: ospf[1009]: %ROUTING-OSPF-3-INTERNALERR : Internal error, path id out of range
Jan 11 03:28:25 rbx-g2-a9.fr.eu 88283: RP/0/RSP0/CPU0:Jan 11 02:28:25 UTC: ospf[1009]: %ROUTING-OSPF-3-INTERNALERR : Internal error, path id out of range
Jan 11 03:28:25 rbx-g2-a9.fr.eu 88284: RP/0/RSP0/CPU0:Jan 11 02:28:25 UTC: ospf[1009]: %ROUTING-OSPF-3-INTERNALERR : Internal error, path id out of range
Jan 11 03:28:25 rbx-g2-a9.fr.eu 88285: RP/0/RSP0/CPU0:Jan 11 02:28:25 UTC: ospf[1009]: %ROUTING-OSPF-3-INTERNALERR : Internal error, path id out of range


Comment by OVH - Thursday, 12 January 2012, 21:27PM

We had to force a switch on the RSP1 and do a rest of the process opsfv3. The restarting of the RSPO is in progress.


Comment by OVH - Thursday, 12 January 2012, 21:28PM

Activation of new packets in progress on G2:

Warning: - the set of packages being activated.
Info: This operation will reload the following nodes in parallel:
Info: 0/RSP0/CPU0 (RP) (SDR: Owner)
Info: 0/RSP1/CPU0 (RP) (SDR: Owner)
Info: 0/0/CPU0 (LC) (SDR: Owner)
Info: 0/1/CPU0 (LC) (SDR: Owner)
Info: 0/2/CPU0 (LC) (SDR: Owner)
Info: 0/3/CPU0 (LC) (SDR: Owner)
Info: 0/4/CPU0 (LC) (SDR: Owner)
Info: 0/5/CPU0 (LC) (SDR: Owner)
Info: 0/6/CPU0 (LC) (SDR: Owner)
Info: 0/7/CPU0 (LC) (SDR: Owner)
Proceed with this install operation (y/n)? [y]
Info: Install Method: Parallel Reload


Comment by OVH - Thursday, 12 January 2012, 21:29PM

The restarting of the router is in progress.


Comment by OVH - Thursday, 12 January 2012, 21:29PM

The router has correctly started on the new images. We are updating the fpd:
FPD upgrade in progress. Max timeout remaining 89 min.


Comment by OVH - Thursday, 12 January 2012, 21:30PM

The upgarde of FPDs is done. We are rebooting a last time on all the nodes.


Comment by OVH - Thursday, 12 January 2012, 21:31PM

The router is updated and is stable. We reactive the trafic on it.


Comment by OVH - Thursday, 12 January 2012, 21:31PM

The trafic is reactivated. The 2 routers rbx-g1 ad g2 are working normally.


Comment by OVH - Tuesday, 17 January 2012, 00:19AM

We will get back to work tonight since 00:00 CET.
The two routers are now updated in soft and we installed the new IOS XR on the new 440 RSPS. We will insert them tonight in the chassis rbx-g1 and g2.
If everything goes well, we can insert new 24x10G.


Comment by OVH - Tuesday, 17 January 2012, 00:21AM

We started the worktasks.
We are moving the traffic to rbx-g2.


Comment by OVH - Tuesday, 17 January 2012, 00:59AM

We are replacing RSP cards.


Comment by OVH - Tuesday, 17 January 2012, 01:02AM

The router has booted. We are repushing its configuration.


Comment by OVH - Tuesday, 17 January 2012, 01:04AM

The router started properly on the new RSPs. We descend the config.


Comment by OVH - Tuesday, 17 January 2012, 02:23AM

g1 has been configured. we will pass it the traffic.


Comment by OVH - Tuesday, 17 January 2012, 02:23AM

Traffic flows normally on g1.We'll start with G2 .


Comment by OVH - Tuesday, 17 January 2012, 02:24AM

We are replacing the RSP on rbx-g2.


Comment by OVH - Tuesday, 17 January 2012, 02:26AM

A new config port console is required ont this RSPS:
line console
exec-timeout 0 0
parity none
stopbits 1
length 60
access-class ingress 10
access-class egress 10


Comment by OVH - Tuesday, 17 January 2012, 02:26AM

We descend the config on g2.


Comment by OVH - Tuesday, 17 January 2012, 02:27AM

We are reactivating traffic on g2.


Comment by OVH - Tuesday, 17 January 2012, 02:28AM

The two routers are stable. We replace the card 0 of rbx-g2 by a new card 24x10G.


Comment by OVH - Tuesday, 17 January 2012, 02:29AM

RP/0/RSP0/CPU0:rbx-g2-a9#RP/0/RSP1/CPU0:Jan 17 01:21:39 UTC: dao_tmp[55]: Card Inserted 0/0/CPU0
RP/0/RSP0/CPU0:Jan 17 01:21:39 UTC: dao_tmp[55]: Card Inserted 0/0/CPU0
RP/0/RSP0/CPU0:Jan 17 01:21:50 UTC: shelfmgr[385]: %PLATFORM-SHELFMGR-3-MIXED_MEMORY_POWERED : Node 0/0/CPU0 is powered, system running mixed li
ode
RP/0/RSP0/CPU0:Jan 17 01:21:50 UTC: shelfmgr[385]: %PLATFORM-SHELFMGR-3-MIXED_MEMORY_ALARM : System enters mixed line card memory mode
RP/0/RSP0/CPU0:Jan 17 01:21:50 UTC: eem_ed_oir[190]: Messge received content : Event 0 NodeId: 0x801


Comment by OVH - Tuesday, 17 January 2012, 04:10AM

Our optics are well recognized by the cards but do not emit light. We suspect there is an incompatibility.


Comment by OVH - Tuesday, 17 January 2012, 04:13AM

We fallback on 8x10G.We will work with Cisco to fix the optics compatibility problem and check the exact implications of the message "MIXED_MEMORY_ALARM" back from the card while inserting.


Comment by OVH - Tuesday, 17 January 2012, 04:15AM

We could finally run another kind of optics.We'll let this 24x10G run in order to check its operating. If there is no more inconveniences, we will insert the next.


Comment by OVH - Thursday, 26 January 2012, 21:18PM

We are going to upgrade the capa between rbx-g1/g2-a9 and rbx-g1/g2-n7. The upgrade on the whole g2 is in progress. On g1, we are going to replace some 8T cards by 24x10G cards. We are going to do this operation tonight starting from 00:00 CET.


Comment by OVH - Friday, 27 January 2012, 01:40AM

We started the intervention in order to replace the cards 8x10G by 24x10G.
We will intervene first on rbx-g2-a9 then rbx-g1-a9.


Comment by OVH - Friday, 27 January 2012, 01:42AM

The new cards are in production on rbx-g2. Same thing for rbx-g1.


Comment by OVH - Friday, 27 January 2012, 01:43AM

The new cards are in production on rbx-g1-a9.