rssLink RSS for all categories
 
icon_red
icon_green
icon_red
icon_red
icon_red
icon_green
icon_green
icon_red
icon_red
icon_red
icon_orange
icon_green
icon_green
icon_green
icon_red
icon_blue
icon_red
icon_orange
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_green
icon_red
icon_orange
icon_green
 

FS#3520 — FS#7462 — 188.165.13/24 188.165.14/24 188.165.15/24 178.33.122/24

Attached to Project— Network
Maintenance
the whole network
CLOSED
100%
Following the update of switch N5 we found a BUG in the newest version that makes sometimes the APR disappear in the network.

We are urgently downgrading to the less newer version.
Date:  Saturday, 13 October 2012, 15:33PM
Reason for closing:  Done
Comment by OVH - Saturday, 13 October 2012, 03:01AM

The problem appears only on 188.165.13/24 but we will downgrande everything we upgraded 2 days earlier.


Comment by OVH - Saturday, 13 October 2012, 03:01AM

Many ports are on the status inactive :

Eth100/1/1 server inactive 589 full 10G --
Eth100/1/2 server inactive 589 full 10G --
Eth100/1/3 server inactive 589 full 10G --
Eth100/1/4 server notconnec 589 full 10G --
Eth100/1/5 server notconnec 589 full 10G --
Eth100/1/6 server inactive 589 full 10G --
Eth100/1/7 server inactive 589 full 10G --
Eth100/1/8 server inactive 589 full 10G --
Eth100/1/9 server inactive 589 full 10G --
Eth100/1/10 server inactive 589 full 10G --
Eth100/1/11 server sfpAbsent 588 full 10G --
Eth100/1/12 server inactive 589 full 10G --
Eth100/1/13 server inactive 589 full 10G --
Eth100/1/14 server inactive 589 full 10G --
Eth100/1/15 server inactive 589 full 10G --
Eth100/1/16 server inactive 589 full 10G --
Eth100/1/17 server inactive 589 full 10G --
Eth100/1/18 server inactive 589 full 10G --
Eth100/1/19 server connected trunk full 10G --
Eth100/1/20 server sfpAbsent trunk full 10G --
Eth100/1/21 server notconnec 588 full 10G --
Eth100/1/22 server connected 588 full 10G --
Eth100/1/23 server inactive 589 full 10G --
Eth100/1/24 server connected trunk full 10G --
Eth100/1/25 server inactive 589 full 10G --
Eth100/1/26 server inactive 589 full 10G --
Eth100/1/27 server inactive 589 full 10G --
Eth100/1/28 server inactive 589 full 10G --
Eth100/1/29 server sfpAbsent 588 full 10G --
Eth100/1/30 server inactive 589 full 10G --
Eth100/1/31 server sfpAbsent 588


Comment by OVH - Saturday, 13 October 2012, 03:02AM

We restarted the fex

same.

We rebooted the system.


Comment by OVH - Saturday, 13 October 2012, 03:02AM

sw-n5-13.248# reload
WARNING: This command will reboot the system
Do you want to continue? (y/n) [n] y


Comment by OVH - Saturday, 13 October 2012, 03:43AM

It's the same. The ports are down


Comment by OVH - Saturday, 13 October 2012, 03:43AM

Replacing a cable. It's the same.


Comment by OVH - Saturday, 13 October 2012, 03:44AM

FEX is replaced physically with a new one and cut electrically.


Comment by OVH - Saturday, 13 October 2012, 03:45AM

The FEX is electrically cut is up but the ports are still down.

We will wait for the spare to start.


Comment by OVH - Saturday, 13 October 2012, 04:05AM

The fex spare starts.It's the same.

So now ..


Comment by OVH - Saturday, 13 October 2012, 04:11AM

We will upload an older software version. We will move from 5.2.1.N1.1b.bin to 5.2.1.N1.1.bin and then switch to 5.1.3.N2.1.bin

We need 5 minutes to put the images on the two N5 and we will update it fastly, then we'll reboot everything on hard with power cut-off for switchs and fex.


Comment by OVH - Saturday, 13 October 2012, 04:12AM

Images are on N5.

We are rebooting all.


Comment by OVH - Saturday, 13 October 2012, 04:17AM

The N5 has booted.The configuration started.

Then the FEX will start booting and will have to be updated, it usually takes 10min by FEX, it is done simultaneously.


Comment by OVH - Saturday, 13 October 2012, 04:17AM

FEX update

Logs:
10/13/2012 04:05:19.324425: Module register received
10/13/2012 04:05:19.325823: Image Version Mismatch
10/13/2012 04:05:19.326266: Registration response sent
10/13/2012 04:05:19.326737: Requesting satellite to download image


Comment by OVH - Saturday, 13 October 2012, 04:18AM

FEX are booting. ports are UP FINALLY !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!


Comment by OVH - Saturday, 13 October 2012, 04:19AM

everything is UP.

CONCLUSION:
The version 5.2.1X is RACTIOACTIVE!!


Comment by OVH - Saturday, 13 October 2012, 04:28AM

In 5 minutes we will move the FEX spare on the FEX which is in the rack and which is correct.


Comment by OVH - Saturday, 13 October 2012, 04:29AM

The ports configuration is lost. We will re-apply it.


Comment by OVH - Saturday, 13 October 2012, 04:45AM

Configuration is applied.Everything is up.

We will set the FEX 105 properly, which was replaced by the spare.
These servers will be down for more 10 minutes.


Comment by OVH - Saturday, 13 October 2012, 04:45AM

FEX update image 5.1.3
Logs:
10/13/2012 04:41:46.636029: Module register received
10/13/2012 04:41:46.637450: Image Version Mismatch
10/13/2012 04:41:46.638126: Registration response sent
10/13/2012 04:41:46.638647: Requesting satellite to download image


Comment by OVH - Saturday, 13 October 2012, 05:12AM

10/13/2012 04:45:59.702382: Image preload successful.
10/13/2012 04:46:00.822397: Deleting route to FEX
10/13/2012 04:46:00.831361: Module disconnected
10/13/2012 04:46:00.833211: Module Offline
10/13/2012 04:46:00.839272: Deleting route to FEX
10/13/2012 04:46:00.847072: Module disconnected
10/13/2012 04:46:00.890047: Offlining Module
10/13/2012 04:46:00.892061: Deleting route to FEX
10/13/2012 04:46:00.899818: Module disconnected
10/13/2012 04:46:00.963837: Offlining Module


Comment by OVH - Saturday, 13 October 2012, 05:13AM

10/13/2012 04:47:14.816521: Module register received
10/13/2012 04:47:14.818478: Registration response sent
10/13/2012 04:47:15.401136: Module Online Sequence
10/13/2012 04:47:19.281549: Module Online


FEX is up. ports are UP.


Comment by OVH - Saturday, 13 October 2012, 05:43AM

The intervention is completed. All ports are UP and all HG are up in the monitoring.

The origin of the problem:
2 days ago we updated the software on some HG switches. tonight ,suddenly the switch said "servers' ports are down."
we first downgranded the software version from 5.2.1b to 5.2.1 because we had yesterday the first signals that b has problems.
finally we had to downgrade it to 5.1.3 and only then all problems has gone.

This is an unusual problem due to software bugs in network equipment that we are using. it is rare, very rare, but it happens.

We are sorry for the trouble.

Affected customers will have the right to 1 free month since the SLA has largely exploded.