OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#7608 — sup2t TCAM programming
Incident Report for Network & Infrastructure
Resolved
In the SY1/2 version there is a bug in
the TCAM programming in the different cards.
The router is installing a route
and then copy it in each TCAM of each card.
Sometimes this copy is not done correctly !!


This gives this looping traceroute :


Host Loss% Last Avg Best Wrst StDev
1. vss-3-6k.fr.eu 0.9% 1.5 20.1 0.4 234.2 48.3
2. rbx-g1-a9.fr.eu 0.0% 1.0 27.3 0.8 435.5 89.4
3. ams-1-6k.nl.eu 78.2% 5.5 7.0 5.4 21.6 4.3
ams-1-6k.nl.eu
ams-1-6k.nl.eu
ams-1-6k.nl.eu
4. tata.as6453.nl.eu 0.0% 15.0 20.4 5.8 492.7 59.5
rbx-g1-a9.fr.eu
5. 4.68.63.41 25.5% 10.4 10.9 10.2 44.0 3.8
6. rbx-g1-a9.fr.eu 0.0% 19.3 18.1 10.3 242.8 32.9
4.69.162.137
7. ae-59-114.ebr1.Amsterdam1.Level3.net 28.2% 10.5 11.6 10.2 22.8 2.6
8. rbx-g1-a9.fr.eu 0.0% 17.2 19.9 9.0 375.5 46.2
ae-1-6.bar1.Brussels1.Level3.net
9. ae-0-11.bar2.Brussels1.Level3.net 30.9% 8.8 12.7 8.8 75.5 12.0
10. rbx-g1-a9.fr.eu 0.0% 9.9 27.8 9.7 490.1 65.5
BRUTELE.car2.Brussels1.Level3.net
11. 212.68.211.17 30.9% 9.1 13.6 9.0 70.5 12.3
ams-1-6k.nl.eu
12. rbx-g1-a9.fr.eu 0.0% 9.2 19.9 9.1 441.7 42.3
212.68.211.138
13. 212.68.211.26 25.7% 10.8 13.9 10.7 45.0 8.4
ams-1-6k.nl.eu
14. 78.129.127.118 0.0% 32.4 20.1 11.4 229.2 24.1
rbx-g1-a9.fr.eu
15. ???
16. rbx-g1-a9.fr.eu 62.4% 36.9 75.2 36.8 487.7 107.3
17. ???
18. rbx-g1-a9.fr.eu 67.6% 43.7 64.8 42.0 483.0 92.1
19. ???
20. rbx-g1-a9.fr.eu 68.5% 48.9 67.8 47.1 429.7 73.7
21. ???
22. rbx-g1-a9.fr.eu 65.7% 52.7 95.4 52.3 510.9 122.8
23. ???
24. rbx-g1-a9.fr.eu 72.8% 58.8 79.0 57.4 389.7 76.6
25. ???
26. rbx-g1-a9.fr.eu 65.7% 62.9 88.2 62.6 435.1 85.7
27. ???


Update(s):

Date: 2012-11-10 20:43:22 UTC
Everything is up.

Date: 2012-11-10 20:08:16 UTC
Some ports were not remounted.
We must interrupt the cards during 10 minutes.
A stupid bug on the opticals.
We have to change them as soon as possible.

Date: 2012-11-10 20:04:54 UTC
It's fixed :

Host Loss% Last Avg Best Wrst StDev
1. vss-3-6k.fr.eu 0.0% 286.7 45.4 0.4 286.7 88.3
2. rbx-g1-a9.fr.eu 0.0% 2.1 17.1 0.7 362.1 75.2
3. ams-1-6k.nl.eu 30.4% 60.2 32.9 5.5 60.2 17.6
ams-1-6k.nl.eu
ams-1-6k.nl.eu
ams-1-6k.nl.eu
4. tata.as6453.nl.eu 0.0% 12.0 14.7 5.9 42.4 8.6
5. 4.68.63.41 0.0% 15.7 21.4 10.3 45.1 11.4
6. 4.69.162.137 0.0% 15.4 17.1 10.4 48.7 9.2
7. ae-59-114.ebr1.Amsterdam1.Level3.net 0.0% 10.3 19.2 10.3 43.4 10.2
8. ae-1-6.bar1.Brussels1.Level3.net 0.0% 9.0 17.2 9.0 70.9 14.8
9. ae-0-11.bar2.Brussels1.Level3.net 0.0% 15.2 19.0 8.9 73.3 15.0
10. BRUTELE.car2.Brussels1.Level3.net 0.0% 30.6 21.5 9.7 68.7 14.2
11. 212.68.211.17 0.0% 15.1 18.8 9.0 60.9 13.8
12. 212.68.211.138 0.0% 29.7 23.5 9.2 94.8 20.7
13. 212.68.211.26 4.3% 14.4 22.9 10.7 72.7 16.9
14. 78.129.127.118 0.0% 15.3 22.8 11.6 46.5 12.9
15. ???


Date: 2012-11-10 20:04:31 UTC
Fed up !

We are going to reload the router.

Date: 2012-11-10 20:03:25 UTC
We see the ip of ams-1-6k which is tied to
the port channel of the card 3. We are reloading the card 3.

ams-1-6k-SUP2T(config)#no power enable module 3


We still see the problem on the card 5

10 minutes:
ams-1-6k-SUP2T(config)#power enable module 3
5 minutes:
ams-1-6k-SUP2T(config)#no power enable module 5


We still see the problem on card 1

10 minutes:
ams-1-6k-SUP2T(config)#power enable module 5
5 minutes:
ams-1-6k-SUP2T(config)#no power enable module 1

le probleme on voit encore sur la carte 4

10 minutes:
ams-1-6k-SUP2T(config)#power enable module 1
5 minutes:
ams-1-6k-SUP2T(config)#no power enable module 4

Card 3 again !

10 minutes:
ams-1-6k-SUP2T(config)#power enable module 4
5 minutes:
ams-1-6k-SUP2T(config)#no power enable module 3
Posted Nov 10, 2012 - 20:00 UTC