rssLink RSS for all categories
 
icon_red
icon_green
icon_red
icon_red
icon_red
icon_green
icon_green
icon_red
icon_red
icon_red
icon_orange
icon_green
icon_green
icon_green
icon_red
icon_blue
icon_red
icon_orange
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_green
icon_red
icon_green
icon_green
 

FS#5745 — FS#9663 — pcc-1a/b-n7

Attached to Project— Dedicated Cloud
Incident
Backend / Core
CLOSED
100%
We have an incident on application of the config on the switches.

"ERROR: Configuration Failed with Error: Failure Returned from Policy Server"
"CEST: %VLAN_MGR-2-CRITICAL_MSG: Switchport Configuration Failed for msgid 0x37f0c9 rrtoken 0x37f0c9"

We are contacting the manufacturer.
Date:  Tuesday, 12 November 2013, 13:24PM
Reason for closing:  Done
Comment by OVH - Friday, 08 November 2013, 19:33PM

We are working with Cisco to fix this problem. The case in now in P1, which is high priority. It's currently not possible to modify the vlans configuration on the 2 core switchs of RBX Dedicated Cloud. We still don't know if this is related to the NXOS upgrade, the OS that runs on these equipments or if it's a problem related to new routing configurations or something else.
There is no impact on the traffic but it's currently not possible to add new resources.


Comment by OVH - Saturday, 09 November 2013, 00:20AM

Cisco has identified the source of errors as the maximum achieved with the last NXOS version.
Following the update of http://status.ovh.co.uk/?do=details&id=5713 , the Nexus 7000 have not integrated totally the new configurations.

A reload of the chassis is necessary to apply the new configurations.
The reload will be done with cisco teams.

This operation is planned for midnight on the night of Friday, November 8th to Saturday 9th of November.


Comment by OVH - Saturday, 09 November 2013, 00:21AM

We deferred this operation to the night of Monday 11th to Tuesday, 12th November at 00:00 CET.


Comment by OVH - Tuesday, 12 November 2013, 02:11AM

Th intervention started , we rebooted pcc-1b-n7.


Comment by OVH - Tuesday, 12 November 2013, 02:15AM

We started to cut the ports gradually on the pcc-1b-n7 in order to isolate the network. This would not have any impact because the traffic is switched in // by pcc-1a-n7.
However, we still lost connectivity to 3 switches N5 access (25, 28 and 29). Then we have reactivated all ports that had been cut in order to resume traffic as soon as possible.
We are working with Cisco to understand what causes this problem.


Comment by OVH - Tuesday, 12 November 2013, 02:19AM

We kept switching with Cisco.
The first cards are completed and all switching went well.

On the latest cards, following losses of connectivity to pcc-106-n5, pcc-107-n5, pcc-108-n5, pcc-109-n5, pcc-116-n5 and pcc-117-n5, we reactivated all ports which had been cut in order to resume traffic as soon as possible.

We are working with Cisco to understand what causes the malfunction that is related to N5 access (25, 28 and 29).


Comment by OVH - Tuesday, 12 November 2013, 02:20AM

We decided to try to isolate the other switch of the pair, the pcc-1a-n7 whose role is "primary" in the vPC pair. This time, we did not have had any problems.
We are currently rebooting the chassis to fix the problem of data structure on the vlans config.


Comment by OVH - Tuesday, 12 November 2013, 02:22AM

The pcc-1a-n7 is up.The reload has effectively set the data structure of the vlan config.
We are gradually remounting ports then we will try again the manipulation on pcc-1b-n7.


Comment by OVH - Tuesday, 12 November 2013, 08:07AM

We rebooted pcc-1b.


Comment by OVH - Tuesday, 12 November 2013, 08:09AM

The pcc-1b rebooted properly. The infrastructure works properly. We will replace a few optics following the reboot.


Comment by OVH - Tuesday, 12 November 2013, 13:24PM

Everything is up.