Update - We have completed the software upgrade on our LA Router 2 from version 20.x to 21.x. We will not monitor Router 2 to see if we see the same memory leak issue or not. We will update you if anything does arise.
Dec 05, 2022 - 21:07 PST
Update - Juniper Networks advised us that they are looking to duplicate the issue in their "lab environment". Over the weekend we've been getting complaints of high ping and packet-loss during peak hours. Management has decided to update Router 2 from version 20.x to 21.x to see if it will resolve the memory leak issue. We are taking this action in the hopes it resolves the issue faster. Rather than waiting days for Juniper to advise us on what to do next. We will plan to update Router 2 within the next couple of hours. If anything does arise, we will be sure to let you know.
Dec 05, 2022 - 20:10 PST
Update - We've been working with Juniper Networks for the past two days to help identify the root cause of our OSPF outages. After hours of troubleshooting the problem with them. They identified the trigger of the outages due to a memory leak on our FPC (Flexible PIC Concentrators) line cards. We believe the issue started when we upgraded the JUNOS software from version 17.x to 20.x on November 10, 2022. The issue then started to build up over time until it started to have issues and began to crash on November 28, 2022. Knowing what triggers the network outages. Juniper suggested to disable some features on our routers in order avoid the recurring outages we've been experiencing since this started.
Juniper is now troubleshooting the memory leak as a possible bug for the latest JUNOS version we are running. Once a fix is implemented we would have a planned maintenance period where we would apply the patch. If the outages keep occurring we would be forced to downgrade our JUNOS back to an older stable version until Juniper finds a resolution to the current bug.
Disabling the features Juniper suggested they are confident that no more outages would occur. Since this is still under investigation we will be keeping this case opened until we have a final resolution for the outage. As always if anything does arise, we will keep you update.
If you have any questions, please feel free to reach out!
Dec 01, 2022 - 03:14 PST
Update - We suffered another network outage on November 30th 2022 lasting from 7:15PM PST to 7:22PM PST. The cause of this outage was due to the same OSPF connection issue between our routers and core switches. As to the reason why it's happening, we are still working with Juniper Networks to find out. The network as of right now is back to normal. We will update you as more information comes in.
Nov 30, 2022 - 19:24 PST
Update - We haven't had any issues/outages since the most recent outage last night. We are still waiting to hear back from Juniper Networks to see why the failure happened. We are still monitoring the network and will take action if anything does arise. We will update you once we have any more information.
Nov 29, 2022 - 19:31 PST
Update - We suffered another network outage on November 28th 2022 lasting from 11:39PM PST to 11:47PM PST. The cause of this outage was due to the same OSPF connection issue between our routers and core switches. As to the reason why it's happening, we are still working with Juniper Networks to find out. The network as of right now is back to normal. We will update you as more information comes in.
Nov 28, 2022 - 23:49 PST
Update - We were not able to find the cause of the outage. Therefore we have reached out to the manufacture, Juniper Networks, to help investigate the cause. So far the network has been fully operational, but will notify you if an issue does arise. We are still monitoring the situation, and will update you once we consider this outage as resolved.
Nov 28, 2022 - 18:55 PST
Monitoring - We fixed the OSPF communication problem between our routers and core switches. The Los Angeles network is now back online and operational. We are now investigating as to what caused this error and will continue to monitor for any further issues. We will update you once we find the cause and consider this outage as resolved.
Nov 28, 2022 - 15:10 PST
Identified - We have identified the issue as being an OSPF communication disconnect between our routers and core switches. We are applying a fix on the problem now. We will update you soon when we have it.
Nov 28, 2022 - 14:26 PST
Investigating - We are currently investigating a network connectivity issue at our Los Angeles location. Stand by for further updates.
Nov 28, 2022 - 14:07 PST