rssLink RSS for all categories
 
icon_red
icon_green
icon_red
icon_red
icon_red
icon_green
icon_green
icon_red
icon_red
icon_red
icon_orange
icon_green
icon_green
icon_green
icon_red
icon_blue
icon_red
icon_orange
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_green
icon_red
icon_blue
icon_green
 

FS#1168 — FS#5175 — backbone update

Attached to Project— Network
Maintenance
the whole network
CLOSED
100%
We are going to update the routers of the backbone
on the last version of the IOS. The aim of this operation is :
to fix all the bugs tied to the IPV6 and then reformat the TCAM CEF
with a specific configuration.

#sh mls cef maximum-routes
FIB TCAM maximum routes :
=======================
Current :-
-------
IPv4 - 911k
MPLS - 1k
IPv6 - 50k
IP multicast - 1k

Then add it on all the router.
no spanning-tree vlan 1-4094
Date:  Friday, 03 June 2011, 04:56AM
Reason for closing:  Done
Comment by OVH - Thursday, 24 February 2011, 13:39PM

nyk-1-6k


Comment by OVH - Thursday, 24 February 2011, 13:41PM

We switch the LINX 224.


Comment by OVH - Thursday, 24 February 2011, 14:54PM

We are switching the London<>New York link.


Comment by OVH - Thursday, 24 February 2011, 22:50PM

rbx-s4-6k one of the 2 rooters of HG.


Comment by OVH - Friday, 25 February 2011, 04:50AM

rbx-s5-6k one of the 2 rooters of HG.


Comment by OVH - Friday, 25 February 2011, 04:51AM

mar-1-6k one of the routers ISP in Marseille.


Comment by OVH - Friday, 25 February 2011, 04:53AM

One of the cards 10G on mar-1-6k died.

Feb 25 03:02:10 GMT: %PM_SCP-SP-1-LCP_FW_ERR: System resetting module 3 to recover from error: Linecard received system exception. Errcode =
2509000001
Feb 25 03:02:10 GMT: %OIR-SP-3-PWRCYCLE: Card in module 3, is being power-cycled 'Off (Module Reset due to exception or user request)'
Feb 25 03:02:10 GMT: %C6KPWR-SP-4-DISABLED: power to module in slot 3 set Off (Module Reset due to exception or user request)
Feb 25 03:03:25 GMT: %PM_SCP-SP-1-LCP_FW_ERR: System resetting module 3 to recover from error: Linecard received system exception. Errcode =
2509000001
Feb 25 03:03:25 GMT: %OIR-SP-3-PWRCYCLE: Card in module 3, is being power-cycled 'Off (Module Reset due to exception or user request)'
Feb 25 03:03:25 GMT: %C6KPWR-SP-4-DISABLED: power to module in slot 3 set Off (Module Reset due to exception or user request)


Comment by OVH - Friday, 25 February 2011, 04:54AM

rbx-s6-6k one of the 2 routers of pCC.


Comment by OVH - Monday, 07 March 2011, 05:19AM

Tonight,we updated 3 routers of p19:
- p19-7-6k one of the 2 routeurs of p19
- p19-52-6k one of the 3 routeurs of shared hosting and OVH
- p19-57-6k one of the 3 routeurs of shared hosting and OVH

During the reboot and updating,the rooting will be ensured with the other routers.

We can not avoid small cuts that are due to the time needed to converge network (to remove the broken roads and return to the roads
that still work). about 30 seconds here and there.


Comment by OVH - Monday, 07 March 2011, 05:19AM

Started
p19-52


Comment by OVH - Monday, 07 March 2011, 05:20AM

p19-52 done,
Now to p19-57.


Comment by OVH - Monday, 07 March 2011, 05:22AM

p19-57 done

the p19-7 will be done another night. we have to securise better another router via p19-2 before starting with p19-7.


Comment by OVH - Saturday, 12 March 2011, 07:52AM

We will update p19-7.


Comment by OVH - Saturday, 12 March 2011, 07:53AM

Done


Comment by OVH - Saturday, 09 April 2011, 05:23AM

After updating the IOS rbx-1-6k et rbx-2-6k are not going well.

We have problems of version mismatch hardware of some cards with the software version. We are going to resolve this problem with TAC cisco.

Meanwhile,a new version of IOS was released and since the router rbx-2-6k planted
http://status.ovh.co.uk/?do=details&id=1327
with a software error,we will certainly update rbx-2-6k and rbx-1-6k with SXI6.
This may allow us to fix another problem that has appeared following the update:every night at 4:00 o'clock rbx-1-6k is full of CPU for a few minutes. This may be related to hardware problem or maybe software.


Comment by OVH - Saturday, 09 April 2011, 05:47AM

rbx-2-6k done


Comment by OVH - Saturday, 09 April 2011, 05:48AM

rbx-1-6k done


Comment by OVH - Saturday, 09 April 2011, 08:15AM

We have a problem of HSRP on HG routers rbx-s3/s4. We'll update IOS on these 2 routers to see if it fixes the problem.


Comment by OVH - Saturday, 09 April 2011, 08:15AM

rbx-s4 done.


Comment by OVH - Saturday, 09 April 2011, 08:17AM

rbx-s3 under way.


Comment by OVH - Saturday, 09 April 2011, 08:20AM

We have a tiny boot problem.We must take off and set back the disk in the additional card.

Initializing ATA monitor library...
monlib.open(): Open Error = -35
loadprog: error - on file open
Exit at the end of BOOT string

rommon 1 > dir disk0:
Command error complete on disk0:
open: read error...requested 0x4 bytes, got 0xffffffff
trouble reading device magic number

rommon 2 > dir disk0:
Initializing ATA monitor library...
Directory of disk0:
[...]
rommon 3 > reset
Initializing ATA monitor library...
Self extracting the image... [OK]


Comment by OVH - Saturday, 09 April 2011, 08:22AM

Done. the HSRP problem is currently fixed.
We shouldn't face it for a long time.
If everything is fine,we will retry GLBP.


Comment by OVH - Friday, 22 April 2011, 07:11AM

vss-1/2/3/4 are updated.


Comment by OVH - Friday, 06 May 2011, 00:59AM

gsw-1/th1-1/lyo-1/mil-1 are remaining. We are going to deal with these routers tonight.


Comment by OVH - Friday, 06 May 2011, 02:03AM

We are starting the intervention on th1-1-6k.


Comment by OVH - Friday, 06 May 2011, 03:19AM

gsw-1 is updated.


Comment by OVH - Wednesday, 11 May 2011, 11:40AM

We will redirect the traffic routed by lyo-1-6k in order to update the router.


Comment by OVH - Wednesday, 11 May 2011, 11:59AM

The updating of lyo-1 was reported.


Comment by OVH - Friday, 20 May 2011, 12:08PM

We are going to update vie-1-6k .


Comment by OVH - Monday, 23 May 2011, 22:53PM

We are going to update pra-1-6k


Comment by OVH - Monday, 23 May 2011, 22:56PM

We are going to update zur-1-6k


Comment by OVH - Monday, 23 May 2011, 22:56PM

done


Comment by OVH - Tuesday, 24 May 2011, 17:00PM

We will update mil-1-6k


Comment by OVH - Tuesday, 24 May 2011, 17:00PM

We will update mad-1-6k


Comment by OVH - Tuesday, 24 May 2011, 17:01PM

We will update var-1-6k.


Comment by OVH - Tuesday, 24 May 2011, 17:02PM

mil-1-6k up and put again in the routing loop.


Comment by OVH - Tuesday, 24 May 2011, 17:03PM

mad-1-6k up and put again in the routing loop.
var-1-6k up and put again in the routing loop.


Comment by OVH - Tuesday, 24 May 2011, 17:03PM

We will update lyo-1-6k


Comment by OVH - Tuesday, 24 May 2011, 17:04PM

lyo-1-6k is up and put again in the routing loop.


Comment by OVH - Wednesday, 01 June 2011, 14:24PM

We will update fra-1-6k


Comment by OVH - Wednesday, 01 June 2011, 14:24PM

Done.

There is still
fra-5
ams-1
ams-5
p19-2

then it will be finished


Comment by OVH - Friday, 03 June 2011, 00:29AM

p19-2-6k


Comment by OVH - Friday, 03 June 2011, 00:29AM

Jun 2 23:04:05 GMT: %EARL-DFC1-2-SWITCH_BUS_IDLE: Switching bus is idle for 5 seconds. The card grant is 0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x46] JA_DR_RI_0_STA_FCO = 0x3
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x85] JA_DR_RI_1_STA_FCO = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x28A] JA_FI_FT_RCV_RATE_SEL = 0xC
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x289] JA_FI_FT_XMIT_SHAPE = 0xFFF
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x4CB] JA_AG_PM_RAM_FULLNESS_0 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x4CC] JA_AG_PM_RAM_FULLNESS_1 = 0x1F
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x4CD] JA_AG_PM_RAM_FULLNESS_2 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x4CE] JA_AG_PM_RAM_FULLNESS_3 = 0x101
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x4F1] JA_AG_RM_RAM_FULLNESS_0 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x4F2] JA_AG_RM_RAM_FULLNESS_1 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x4F3] JA_AG_RM_RAM_FULLNESS_2 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x4F4] JA_AG_RM_RAM_FULLNESS_3 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x5F6] JA_TM_HI_FULLNESS = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x5F7] JA_TM_LO_FULLNESS = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x46] JA_DR_RI_0_STA_FCO = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x85] JA_DR_RI_1_STA_FCO = 0x3
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x28A] JA_FI_FT_RCV_RATE_SEL = 0xC
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x289] JA_FI_FT_XMIT_SHAPE = 0xFFF
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x4CB] JA_AG_PM_RAM_FULLNESS_0 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x4CC] JA_AG_PM_RAM_FULLNESS_1 = 0x101
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x4CD] JA_AG_PM_RAM_FULLNESS_2 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x4CE] JA_AG_PM_RAM_FULLNESS_3 = 0x102
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x4F1] JA_AG_RM_RAM_FULLNESS_0 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x4F2] JA_AG_RM_RAM_FULLNESS_1 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x4F3] JA_AG_RM_RAM_FULLNESS_2 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x4F4] JA_AG_RM_RAM_FULLNESS_3 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x5F6] JA_TM_HI_FULLNESS = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x5F7] JA_TM_LO_FULLNESS = 0x0


Card #1 is dead.


Comment by OVH - Friday, 03 June 2011, 01:25AM

fra-5-6k in progress.


Comment by OVH - Friday, 03 June 2011, 01:31AM

Tonight we have plenty of problems.
The second update is not doing good with the g1 ,wa have 3 problems to deal with within less than 1 hours.

The router here will not reboot without the intervention of sysadmin. Long live serie cables !

Jun 2 23:32:30 GMT: %OIR-SP-6-CONSOLE: Changing console ownership to switch processor
Jun 2 23:32:30 GMT: %SYS-SP-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.
Jun 2 23:32:33 GMT: %SYS-SP-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.
Jun 2 23:32:33 GMT: %SYS-SP-5-RELOAD: Reload requested by Delayed Reload. Reload Reason: Reload Command.
Jun 2 23:32:33 GMT: %OIR-SP-6-CONSOLE: Changing console ownership to switch processor
monitor: command "boot" aborted due to user interrupt
Exit at the end of BOOT string
rommon 1 > boot
Please reset before booting
rommon 2 > reset
System Bootstrap, Version 8.5(2)
Copyright (c) 1994-2007 by cisco Systems, Inc.
Cat6k-Sup720/SP processor with 1048576 Kbytes of main memory

[...]


Comment by OVH - Friday, 03 June 2011, 01:34AM

fra-5-6k stabilized.

ams-1-6k in progress.


Comment by OVH - Friday, 03 June 2011, 01:34AM

bor-1-6k en in progress.


Comment by OVH - Friday, 03 June 2011, 04:55AM

still ams-5-6k


Comment by OVH - Friday, 03 June 2011, 04:55AM

it's done. ouff ! :)