rssLink RSS for all categories
 
icon_red
icon_green
icon_red
icon_red
icon_red
icon_green
icon_green
icon_red
icon_red
icon_red
icon_orange
icon_green
icon_green
icon_green
icon_red
icon_blue
icon_red
icon_orange
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_green
icon_red
icon_green
icon_green
 

FS#8231 — FS#12102 — Brussels 's optic transmission node - 26/11

Attached to Project— Network
Maintenance
the whole network
CLOSED
100%
Tonight We are going to work on the Brussels POP. Due to a reorganization of this building, we'll move the entire optical transmission node dwdm Infinera in another room.

This node is on routes Roubaix <> Amsterdam and Roubaix <> Frankfurt (and soon Gravelines <> Frankfurt). All of these routes are protected, there is not any impact to service expected for these routes.The only traffic that will be impacted will be dropped traffic locally at Brussels. The traffic will be rerouted to BNIX for the duration of the work (thus changing route but not interrupting service). The voice with ibnterco Belgacom will be unavailable and we apologize.

Work will start at 9pm. The impact on Brussels traffic will start from approximately 10pm.
Date:  Thursday, 27 November 2014, 15:05PM
Reason for closing:  Done
Comment by OVH - Thursday, 27 November 2014, 01:46AM

Our team is on site and prepares the intervention with those of Interoute and Level3. We of course are doing everything possible to reduce downtime to the minimum time.

We currently have approximately 1 hour of delay on the planned schedule. The last preparations are in progress.


Comment by OVH - Thursday, 27 November 2014, 01:47AM

We will reroute BNIX traffic to other paths.


Comment by OVH - Thursday, 27 November 2014, 01:51AM

Optical lines on Brussels just been disconnected. IPs equipment in Brussels are totally isolated.Moving the main chassis Infinera began.


Comment by OVH - Thursday, 27 November 2014, 08:08AM

We have issue there. Work is progressing but not as fast as expected because of various problems: power supply to the new locations, mechanical problems with windows, cleaning of fibers longer than expected (each connector should be linked to fiberscope,kind of electronic microscope, to validate the quality).

We have 2 chassis on 3 active and 2 lines of 4 up (to Amsterdam and Roubaix on coherent 100G). The big issue is behind us and we think we can mount the chassis #3 and the 2nd line to Roubaix by about 1h/1h30. This will be the moment when we can put online the Brussels router.


Comment by OVH - Thursday, 27 November 2014, 10:06AM

Chassis # 3 is being started. Its move has taken longer than expected. In a few minutes, the cards will start to boot. The remaining steps in order to restore service are:
- Reconnect the second optical line to Roubaix
- Start at least one of the 2 cards transponders serving the Brussels POP
- Replace the line
- Reconnect 10G circuits Roubaix <> Brussels

The time to complete these steps depends a lot on any issues encountered but we believe we can restore service in Brussels in 1 hour if there are no new major problem. We have already gone far beyond the maintenance window scheduled and we apologize. Our on-site team is doing everything possible to complete the maintenance as quickly as possible. We will continue to keep you informed of the progress of the work.


Comment by OVH - Thursday, 27 November 2014, 10:06AM

Status at 8;24:

- reconnection of the second optics to Roubaix DONE
- Start at least one of the 2 transponder cards serving the Brussels POP DONE
- Turn up the line DONE
- Reconnect 10G circuits Roubaix <> Brussels DOING


Comment by OVH - Thursday, 27 November 2014, 10:06AM

Status at 8:36:

- Reconnection of 10G circuits Roubaix <> Brussels IN PROGRESS -> 20G out of 40G cappa are UP. The router and all IP equipment for Brussels are online again.


Comment by OVH - Thursday, 27 November 2014, 10:32AM

All the capa to Brussels aer now operational. Only the line to Frankfurt is down after having problems with the migration of fiber pair on location.

The optical node is ready, and is we are lighting the fiber but we suspect one DLM (line module) to Frankfurt is defective. We are working with Infinera to confirm this hypothesis. If this is the case, it must be replaced.

Because of this work, we still have 2x20G down between Roubaix and Strasbourg (since these are circuits that pass through Frankfurt). The remaining links are congested and thus internal traffic Roubaix <> Strasbourg is currently degraded (packet loss).

The local team has been working hard and nonstop since 12am (not including preparation time). We are sending relief from Roubaix in case the resolution of remaining issues require an extended stay at the site.


Comment by OVH - Thursday, 27 November 2014, 10:33AM

We have reestablished IP traffic on BNIX.


Comment by OVH - Thursday, 27 November 2014, 10:37AM

We've fixed the problem with the DLM card.

The cable issue at the level of the fiber pair to Frankfort has been identifies. We will aligned here in a few minutes.


Comment by OVH - Thursday, 27 November 2014, 15:04PM

Finally, we had a battle against time and the alignment of the latest fiber to Frankfurt lasted almost 3 hours due to wiring problems between the building and the neighboring level3 data center through which the fibers must pass through. The managers of the various buildings and the fiber provider have therefore worked to check each patch panel throughly.
The clock stopped at 3pm for the onsite intervention by our ground team. Congratulations to them.

The optical node is now operating normally. The 40G capa is missing between Roubaix and Strasbourg now all is operational.