Get webhook notifications whenever Network & Infrastructure creates an incident, updates an incident, resolves an incident or changes a component status.
We are going to update the routers of the backbone
on the last version of the IOS. The aim of this operation is :
to fix all the bugs tied to the IPV6 and then reformat the TCAM CEF
with a specific configuration.
#sh mls cef maximum-routes
FIB TCAM maximum routes :
=======================
Current :-
-------
IPv4 - 911k
MPLS - 1k
IPv6 - 50k
IP multicast - 1k
Then add it on all the router.
no spanning-tree vlan 1-4094
Update(s):
Date: 2011-06-03 02:55:52 UTC it's done. ouff ! :)
Date: 2011-06-03 02:55:28 UTC still ams-5-6k
Date: 2011-06-02 23:34:57 UTC bor-1-6k en in progress.
Date: 2011-06-02 23:34:35 UTC fra-5-6k stabilized.
ams-1-6k in progress.
Date: 2011-06-02 23:31:29 UTC Tonight we have plenty of problems.
The second update is not doing good with the g1 ,wa have 3 problems to deal with within less than 1 hours.
The router here will not reboot without the intervention of sysadmin. Long live serie cables !
Jun 2 23:32:30 GMT: %OIR-SP-6-CONSOLE: Changing console ownership to switch processor
Jun 2 23:32:30 GMT: %SYS-SP-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.
Jun 2 23:32:33 GMT: %SYS-SP-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.
Jun 2 23:32:33 GMT: %SYS-SP-5-RELOAD: Reload requested by Delayed Reload. Reload Reason: Reload Command.
Jun 2 23:32:33 GMT: %OIR-SP-6-CONSOLE: Changing console ownership to switch processor
monitor: command \"boot\" aborted due to user interrupt
Exit at the end of BOOT string
rommon 1 > boot
Please reset before booting
rommon 2 > reset
System Bootstrap, Version 8.5(2)
Copyright (c) 1994-2007 by cisco Systems, Inc.
Cat6k-Sup720/SP processor with 1048576 Kbytes of main memory
[...]
Date: 2011-06-02 23:25:33 UTC fra-5-6k in progress.
Date: 2011-06-02 22:29:59 UTC Jun 2 23:04:05 GMT: %EARL-DFC1-2-SWITCH_BUS_IDLE: Switching bus is idle for 5 seconds. The card grant is 0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x46] JA_DR_RI_0_STA_FCO = 0x3
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x85] JA_DR_RI_1_STA_FCO = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x28A] JA_FI_FT_RCV_RATE_SEL = 0xC
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x289] JA_FI_FT_XMIT_SHAPE = 0xFFF
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x4CB] JA_AG_PM_RAM_FULLNESS_0 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x4CC] JA_AG_PM_RAM_FULLNESS_1 = 0x1F
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x4CD] JA_AG_PM_RAM_FULLNESS_2 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x4CE] JA_AG_PM_RAM_FULLNESS_3 = 0x101
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x4F1] JA_AG_RM_RAM_FULLNESS_0 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x4F2] JA_AG_RM_RAM_FULLNESS_1 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x4F3] JA_AG_RM_RAM_FULLNESS_2 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x4F4] JA_AG_RM_RAM_FULLNESS_3 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x5F6] JA_TM_HI_FULLNESS = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x5F7] JA_TM_LO_FULLNESS = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x46] JA_DR_RI_0_STA_FCO = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x85] JA_DR_RI_1_STA_FCO = 0x3
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x28A] JA_FI_FT_RCV_RATE_SEL = 0xC
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x289] JA_FI_FT_XMIT_SHAPE = 0xFFF
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x4CB] JA_AG_PM_RAM_FULLNESS_0 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x4CC] JA_AG_PM_RAM_FULLNESS_1 = 0x101
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x4CD] JA_AG_PM_RAM_FULLNESS_2 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x4CE] JA_AG_PM_RAM_FULLNESS_3 = 0x102
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x4F1] JA_AG_RM_RAM_FULLNESS_0 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x4F2] JA_AG_RM_RAM_FULLNESS_1 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x4F3] JA_AG_RM_RAM_FULLNESS_2 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x4F4] JA_AG_RM_RAM_FULLNESS_3 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x5F6] JA_TM_HI_FULLNESS = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x5F7] JA_TM_LO_FULLNESS = 0x0
Card #1 is dead.
Date: 2011-06-02 22:29:10 UTC p19-2-6k
Date: 2011-06-01 12:24:54 UTC Done.
There is still
fra-5
ams-1
ams-5
p19-2
then it will be finished
Date: 2011-06-01 12:24:12 UTC We will update fra-1-6k
Date: 2011-05-24 15:04:25 UTC lyo-1-6k is up and put again in the routing loop.
Date: 2011-05-24 15:03:38 UTC We will update lyo-1-6k
Date: 2011-05-24 15:03:11 UTC mad-1-6k up and put again in the routing loop.
var-1-6k up and put again in the routing loop.
Date: 2011-05-24 15:02:38 UTC mil-1-6k up and put again in the routing loop.
Date: 2011-05-24 15:01:01 UTC We will update var-1-6k.
Date: 2011-05-24 15:00:37 UTC We will update mad-1-6k
Date: 2011-05-24 15:00:07 UTC We will update mil-1-6k
Date: 2011-05-23 20:56:31 UTC done
Date: 2011-05-23 20:56:05 UTC We are going to update zur-1-6k
Date: 2011-05-23 20:53:13 UTC We are going to update pra-1-6k
Date: 2011-05-20 10:08:34 UTC We are going to update vie-1-6k .
Date: 2011-05-11 09:59:05 UTC The updating of lyo-1 was reported.
Date: 2011-05-11 09:40:21 UTC We will redirect the traffic routed by lyo-1-6k in order to update the router.
Date: 2011-05-06 01:19:02 UTC gsw-1 is updated.
Date: 2011-05-06 00:03:32 UTC We are starting the intervention on th1-1-6k.
Date: 2011-05-05 22:59:10 UTC gsw-1/th1-1/lyo-1/mil-1 are remaining. We are going to deal with these routers tonight.
Date: 2011-04-22 05:11:21 UTC vss-1/2/3/4 are updated.
Date: 2011-04-09 06:22:46 UTC Done. the HSRP problem is currently fixed.
We shouldn't face it for a long time.
If everything is fine,we will retry GLBP.
Date: 2011-04-09 06:20:40 UTC We have a tiny boot problem.We must take off and set back the disk in the additional card.
Initializing ATA monitor library...
monlib.open(): Open Error = -35
loadprog: error - on file open
Exit at the end of BOOT string
rommon 1 > dir disk0:
Command error complete on disk0:
open: read error...requested 0x4 bytes, got 0xffffffff
trouble reading device magic number
rommon 2 > dir disk0:
Initializing ATA monitor library...
Directory of disk0:
[...]
rommon 3 > reset
Initializing ATA monitor library...
Self extracting the image... [OK]
Date: 2011-04-09 06:17:13 UTC rbx-s3 under way.
Date: 2011-04-09 06:15:44 UTC rbx-s4 done.
Date: 2011-04-09 06:15:20 UTC We have a problem of HSRP on HG routers rbx-s3/s4. We'll update IOS on these 2 routers to see if it fixes the problem.
Date: 2011-04-09 03:48:09 UTC rbx-1-6k done
Date: 2011-04-09 03:47:55 UTC rbx-2-6k done
Date: 2011-04-09 03:23:56 UTC After updating the IOS rbx-1-6k et rbx-2-6k are not going well.
We have problems of version mismatch hardware of some cards with the software version. We are going to resolve this problem with TAC cisco.
Meanwhile,a new version of IOS was released and since the router rbx-2-6k planted
http://status.ovh.co.uk/?do=details&id=1327 with a software error,we will certainly update rbx-2-6k and rbx-1-6k with SXI6.
This may allow us to fix another problem that has appeared following the update:every night at 4:00 o'clock rbx-1-6k is full of CPU for a few minutes. This may be related to hardware problem or maybe software.
Date: 2011-03-12 06:53:07 UTC Done
Date: 2011-03-12 06:52:59 UTC We will update p19-7.
Date: 2011-03-07 04:22:08 UTC p19-57 done
the p19-7 will be done another night. we have to securise better another router via p19-2 before starting with p19-7.
Date: 2011-03-07 04:20:00 UTC p19-52 done,
Now to p19-57.
Date: 2011-03-07 04:19:37 UTC Started
p19-52
Date: 2011-03-07 04:19:16 UTC Tonight,we updated 3 routers of p19:
- p19-7-6k one of the 2 routeurs of p19
- p19-52-6k one of the 3 routeurs of shared hosting and OVH
- p19-57-6k one of the 3 routeurs of shared hosting and OVH
During the reboot and updating,the rooting will be ensured with the other routers.
We can not avoid small cuts that are due to the time needed to converge network (to remove the broken roads and return to the roads
that still work). about 30 seconds here and there.
Date: 2011-02-25 03:54:03 UTC rbx-s6-6k one of the 2 routers of pCC.
Date: 2011-02-25 03:53:25 UTC One of the cards 10G on mar-1-6k died.
Feb 25 03:02:10 GMT: %PM_SCP-SP-1-LCP_FW_ERR: System resetting module 3 to recover from error: Linecard received system exception. Errcode =
2509000001
Feb 25 03:02:10 GMT: %OIR-SP-3-PWRCYCLE: Card in module 3, is being power-cycled 'Off (Module Reset due to exception or user request)'
Feb 25 03:02:10 GMT: %C6KPWR-SP-4-DISABLED: power to module in slot 3 set Off (Module Reset due to exception or user request)
Feb 25 03:03:25 GMT: %PM_SCP-SP-1-LCP_FW_ERR: System resetting module 3 to recover from error: Linecard received system exception. Errcode =
2509000001
Feb 25 03:03:25 GMT: %OIR-SP-3-PWRCYCLE: Card in module 3, is being power-cycled 'Off (Module Reset due to exception or user request)'
Feb 25 03:03:25 GMT: %C6KPWR-SP-4-DISABLED: power to module in slot 3 set Off (Module Reset due to exception or user request)
Date: 2011-02-25 03:51:27 UTC mar-1-6k one of the routers ISP in Marseille.
Date: 2011-02-25 03:50:44 UTC rbx-s5-6k one of the 2 rooters of HG.
Date: 2011-02-24 21:50:12 UTC rbx-s4-6k one of the 2 rooters of HG.
Date: 2011-02-24 13:54:29 UTC We are switching the London<>New York link.
Date: 2011-02-24 12:41:20 UTC We switch the LINX 224.