May 29, 2024 5:51PM EDT
[Monitoring] Update #6 - Development activities continue, with a target date of June 13, 2024 for production implementation. IT Lab Ops continues to check daily for stuck accessions; none found since Thursday, May 23.
These updates will be paused until the maintenance is complete on June 13, unless new information or issues arise.
Update #5 - Development activities still on track and expected to be ready for production late next week.
IT Lab Ops continues to check daily for orders stuck in LYNXX. No accessions have been found since Thursday, May 23.
Update #4: Items continue on track.
IT Lab Ops will continue to perform the manual process to check for orders stuck in LYNXX. This process will be performed 5:00 – 6:00 p.m. EDT, over the weekend/US holiday.
Update #3: The manual procedure to check for orders stuck in LYNXX and re-release them, remains in place.
A monitoring process for accessions released in LYNXX but partial/unreleased in VC+ is still under assessment. These continue to be escalated to Customer Service and Lab Liaisons.
IT Teams are making progress on a permanent fix for these issues.
Update #2:
IT Teams continue to evaluate options for a monitoring process to identify accessions that are released in LYNXX but are partial/unreleased in VC+.
Progress on permanent solutions for these issues continue.
Update #1:
For Orders stuck in LYNXX and partial in VC+:
IT is running a manual procedure to check for accessions and re-release them. This process will continue to be ran at 5:00 – 6:00 p.m. EDT and 7:00 – 8:00 a.m. EDT, for the rest of the week and then once a day over the weekend/US holiday.
For Orders released in LYNXX and partial/unreleased in VC+:
IT is working on a monitoring process to identify these accessions, in the meantime, these will continue to be escalated to Customer Service and Lab Liaisons.
Teams are currently working on a permanent fix for these issues.
Severity: Medium
Incident Start Date and Time: May 17th 9:30 am ET (estimated)
Incident Statement: This is a continuation of the IT Notification issued and resolved last week. We are still seeing a spike in escalations to Customer Support and the Lab Liaisons to re-release accessions which are not syncing properly from LYNXX to VC+.
Two issues have been identified and being worked on by teams.
Orders are showing as stuck in LYNXX and partial/unreleased in VC+. IT has instituted a manual process to check for these accessions and re-release them two times each day. Run times are 5:00 – 6:00 p.m. EDT and 7:00 – 8:00 a.m. EDT. This will continue daily until a permanent fix is in place.
Orders are showing as expected in LYNXX and partial/unreleased in VC+. Currently there is not a way to identify these as they show completely resulted in LYNXX, so these will continue to be escalated to Customer Service and Lab Liaisons.
Groups and Regions Impacted: US and Canada Customer Support
Application Impacted: LYNXX
Business Impact: Some results are not fully releasing through our ecosystem, causing increased customer calls that require manual re-rereleasing.
Resolution Teams: LYNXX Developers, LYNXX SRE, RDE Developers, Enterprise Architecture, Middleware, VC+ Team, Enterprise IT Lab Ops.
Next Update: Thursday, June 13, 2024 upon maintenance completion.
May 28, 2024 4:17PM EDT
[Monitoring] Update #5 - Development activities still on track and expected to be ready for production late next week.
IT Lab Ops continues to check daily for orders stuck in LYNXX. No accessions have been found since Thursday, May 23.
Update #4: Items continue on track.
IT Lab Ops will continue to perform the manual process to check for orders stuck in LYNXX. This process will be performed 5:00 – 6:00 p.m. EDT, over the weekend/US holiday.
Update #3: The manual procedure to check for orders stuck in LYNXX and re-release them, remains in place.
A monitoring process for accessions released in LYNXX but partial/unreleased in VC+ is still under assessment. These continue to be escalated to Customer Service and Lab Liaisons.
IT Teams are making progress on a permanent fix for these issues.
Update #2:
IT Teams continue to evaluate options for a monitoring process to identify accessions that are released in LYNXX but are partial/unreleased in VC+.
Progress on permanent solutions for these issues continue.
Update #1:
For Orders stuck in LYNXX and partial in VC+:
IT is running a manual procedure to check for accessions and re-release them. This process will continue to be ran at 5:00 – 6:00 p.m. EDT and 7:00 – 8:00 a.m. EDT, for the rest of the week and then once a day over the weekend/US holiday.
For Orders released in LYNXX and partial/unreleased in VC+:
IT is working on a monitoring process to identify these accessions, in the meantime, these will continue to be escalated to Customer Service and Lab Liaisons.
Teams are currently working on a permanent fix for these issues.
Severity: Medium
Incident Start Date and Time: May 17th 9:30 am ET (estimated)
Incident Statement: This is a continuation of the IT Notification issued and resolved last week. We are still seeing a spike in escalations to Customer Support and the Lab Liaisons to re-release accessions which are not syncing properly from LYNXX to VC+.
Two issues have been identified and being worked on by teams.
Orders are showing as stuck in LYNXX and partial/unreleased in VC+. IT has instituted a manual process to check for these accessions and re-release them two times each day. Run times are 5:00 – 6:00 p.m. EDT and 7:00 – 8:00 a.m. EDT. This will continue daily until a permanent fix is in place.
Orders are showing as expected in LYNXX and partial/unreleased in VC+. Currently there is not a way to identify these as they show completely resulted in LYNXX, so these will continue to be escalated to Customer Service and Lab Liaisons.
Groups and Regions Impacted: US and Canada Customer Support
Application Impacted: LYNXX
Business Impact: Some results are not fully releasing through our ecosystem, causing increased customer calls that require manual re-rereleasing.
Resolution Teams: LYNXX Developers, LYNXX SRE, RDE Developers, Enterprise Architecture, Middleware, VC+ Team, Enterprise IT Lab Ops.
Next Update: Wednesday, May 29, 4:00 p.m. EDT
May 24, 2024 8:03PM EDT
[Identified] Update #4:
Items continue on track.
IT Lab Ops will continue to perform the manual process to check for orders stuck in LYNXX. This process will be performed 5:00 – 6:00 p.m. EDT, over the weekend/US holiday.
Update #3:
The manual procedure to check for orders stuck in LYNXX and re-release them, remain in place.
A monitoring process for accessions released in LYNXX but partial/unreleased in VC+ is still under assessment. These continue to be escalated to Customer Service and Lab Liaisons.
IT Teams are making progress on a permanent fix for these issues.
Update #2:
IT Teams continue to evaluate options for a monitoring process to identify accessions that are released in LYNXX but are partial/unreleased in VC+.
Progress on permanent solutions for these issues continue.
Update #1:
For Orders stuck in LYNXX and partial in VC+:
IT is running a manual procedure to check for accessions and re-release them. This process will continue to be ran at 5:00 – 6:00 p.m. EDT and 7:00 – 8:00 a.m. EDT, for the rest of the week and then once a day over the weekend/US holiday.
For Orders released in LYNXX and partial/unreleased in VC+:
IT is working on a monitoring process to identify these accessions, in the meantime, these will continue to be escalated to Customer Service and Lab Liaisons.
Teams are currently working on a permanent fix for these issues.
Severity: Medium
Incident Start Date and Time: May 17th 9:30 am ET (estimated)
Incident Statement: This is a continuation of the IT Notification issued and resolved last week. We are still seeing a spike in escalations to Customer Support and the Lab Liaisons to re-release accessions which are not syncing properly from LYNXX to VC+.
Two issues have been identified and being worked on by teams.
Orders are showing as stuck in LYNXX and partial/unreleased in VC+. IT has instituted a manual process to check for these accessions and re-release them two times each day. Run times are 5:00 – 6:00 p.m. EDT and 7:00 – 8:00 a.m. EDT. This will continue daily until a permanent fix is in place.
Orders are showing as expected in LYNXX and partial/unreleased in VC+. Currently there is not a way to identify these as they show completely resulted in LYNXX, so these will continue to be escalated to Customer Service and Lab Liaisons.
Groups and Regions Impacted: US and Canada Customer Support
Application Impacted: LYNXX
Business Impact: Some results are not fully releasing through our ecosystem, causing increased customer calls that require manual re-rereleasing.
Resolution Teams: LYNXX Developers, LYNXX SRE, RDE Developers, Enterprise Architecture, Middleware, VC+ Team, Enterprise IT Lab Ops.
Next Update: Tuesday, May 28, 10:00 a.m. EDT
May 24, 2024 12:35PM EDT
[Identified] Update #3:
The manual procedure to check for orders stuck in LYNXX and re-release them, remain in place.
A monitoring process for accessions released in LYNXX but partial/unreleased in VC+ is still under assessment. These continue to be escalated to Customer Service and Lab Liaisons.
IT Teams are making progress on a permanent fix for these issues.
Update #2:
IT Teams continue to evaluate options for a monitoring process to identify accessions that are released in LYNXX but are partial/unreleased in VC+.
Progress on permanent solutions for these issues continue.
Update #1:
For Orders stuck in LYNXX and partial in VC+:
IT is running a manual procedure to check for accessions and re-release them. This process will continue to be ran at 5:00 – 6:00 p.m. EDT and 7:00 – 8:00 a.m. EDT, for the rest of the week and then once a day over the weekend/US holiday.
For Orders released in LYNXX and partial/unreleased in VC+:
IT is working on a monitoring process to identify these accessions, in the meantime, these will continue to be escalated to Customer Service and Lab Liaisons.
Teams are currently working on a permanent fix for these issues.
Severity: Medium
Incident Start Date and Time: May 17th 9:30 am ET (estimated)
Incident Statement: This is a continuation of the IT Notification issued and resolved last week. We are still seeing a spike in escalations to Customer Support and the Lab Liaisons to re-release accessions which are not syncing properly from LYNXX to VC+.
Two issues have been identified and being worked on by teams.
Orders are showing as stuck in LYNXX and partial/unreleased in VC+. IT has instituted a manual process to check for these accessions and re-release them two times each day. Run times are 5:00 – 6:00 p.m. EDT and 7:00 – 8:00 a.m. EDT. This will continue daily until a permanent fix is in place.
Orders are showing as expected in LYNXX and partial/unreleased in VC+. Currently there is not a way to identify these as they show completely resulted in LYNXX, so these will continue to be escalated to Customer Service and Lab Liaisons.
Groups and Regions Impacted: US and Canada Customer Support
Application Impacted: LYNXX
Business Impact: Some results are not fully releasing through our ecosystem, causing increased customer calls that require manual re-rereleasing.
Resolution Teams: LYNXX Developers, LYNXX SRE, RDE Developers, Enterprise Architecture, Middleware, VC+ Team, Enterprise IT Lab Ops.
Next Update: Friday, May 24, 4:00 p.m. EDT
May 23, 2024 4:12PM EDT
[Identified] Update #2:
IT Teams continue to evaluate options for a monitoring process to identify accessions that are released in LYNXX but are partial/unreleased in VC+.
Progress on permanent solutions for these issues continue.
Update #1:
For Orders stuck in LYNXX and partial in VC+:
IT is running a manual procedure to check for accessions and re-release them. This process will continue to be ran at 5:00 – 6:00 p.m. EDT and 7:00 – 8:00 a.m. EDT, for the rest of the week and then once a day over the weekend/US holiday.
For Orders released in LYNXX and partial/unreleased in VC+:
IT is working on a monitoring process to identify these accessions, in the meantime, these will continue to be escalated to Customer Service and Lab Liaisons.
Teams are currently working on a permanent fix for these issues.
Severity: Medium
Incident Start Date and Time: May 17th 9:30 am ET (estimated)
Incident Statement: This is a continuation of the IT Notification issued and resolved last week. We are still seeing a spike in escalations to Customer Support and the Lab Liaisons to re-release accessions which are not syncing properly from LYNXX to VC+.
Two issues have been identified and being worked on by teams.
Orders are showing as stuck in LYNXX and partial/unreleased in VC+. IT has instituted a manual process to check for these accessions and re-release them two times each day. Run times are 5:00 – 6:00 p.m. EDT and 7:00 – 8:00 a.m. EDT. This will continue daily until a permanent fix is in place.
Orders are showing as expected in LYNXX and partial/unreleased in VC+. Currently there is not a way to identify these as they show completely resulted in LYNXX, so these will continue to be escalated to Customer Service and Lab Liaisons.
Groups and Regions Impacted: US and Canada Customer Support
Application Impacted: LYNXX
Business Impact: Some results are not fully releasing through our ecosystem, causing increased customer calls that require manual re-rereleasing.
Resolution Teams: LYNXX Developers, LYNXX SRE, RDE Developers, Enterprise Architecture, Middleware, VC+ Team, Enterprise IT Lab Ops.
Next Update: Friday, May 24, 4:00 p.m. EDT
May 23, 2024 10:02AM EDT
[Identified] Update #1:
For Orders stuck in LYNXX and partial in VC+:
IT is running a manual procedure to check for accessions and re-release them. This process will continue to be ran at 5:00 – 6:00 p.m. EDT and 7:00 – 8:00 a.m. EDT, for the rest of the week and then once a day over the weekend/US holiday.
For Orders released in LYNXX and partial/unreleased in VC+:
IT is working on a monitoring process to identify these accessions, in the meantime, these will continue to be escalated to Customer Service and Lab Liaisons.
Teams are currently working on a permanent fix for these issues.
Severity: Medium
Incident Start Date and Time: May 17th 9:30 am ET (estimated)
Incident Statement: This is a continuation of the IT Notification issued and resolved last week. We are still seeing a spike in escalations to Customer Support and the Lab Liaisons to re-release accessions which are not syncing properly from LYNXX to VC+.
Two issues have been identified and being worked on by teams.
Orders are showing as stuck in LYNXX and partial/unreleased in VC+. IT has instituted a manual process to check for these accessions and re-release them two times each day. Run times are 5:00 – 6:00 p.m. EDT and 7:00 – 8:00 a.m. EDT. This will continue daily until a permanent fix is in place.
Orders are showing as expected in LYNXX and partial/unreleased in VC+. Currently there is not a way to identify these as they show completely resulted in LYNXX, so these will continue to be escalated to Customer Service and Lab Liaisons.
Groups and Regions Impacted: US and Canada Customer Support
Application Impacted: LYNXX
Business Impact: Some results are not fully releasing through our ecosystem, causing increased customer calls that require manual re-rereleasing.
Resolution Teams: LYNXX Developers, LYNXX SRE, RDE Developers, Enterprise Architecture, Middleware, VC+ Team, Enterprise IT Lab Ops.
Next Update: Thursday, May 23, 4:00 p.m. EDT
May 22, 2024 4:29PM EDT
[Identified] Severity: Medium
Incident Start Date and Time: May 17th 9:30 am ET (estimated)
Incident Statement: This is a continuation of the IT Notification issued and resolved last week. We are still seeing a spike in escalations to Customer Support and the Lab Liaisons to re-release accessions which are not syncing properly from LYNXX to VC+.
Two issues have been identified and being worked on by teams.
Orders are showing as stuck in LYNXX and partial/unreleased in VC+. IT has instituted a manual process to check for these accessions and re-release them two times each day. Run times are 5:00 – 6:00 p.m. EDT and 7:00 – 8:00 a.m. EDT. This will continue daily until a permanent fix is in place.
Orders are showing as expected in LYNXX and partial/unreleased in VC+. Currently there is not a way to identify these as they show completely resulted in LYNXX, so these will continue to be escalated to Customer Service and Lab Liaisons.
Groups and Regions Impacted: US and Canada Customer Support
Application Impacted: LYNXX
Business Impact: Some results are not fully releasing through our ecosystem, causing increased customer calls that require manual re-rereleasing.
Resolution Teams: LYNXX Developers, LYNXX SRE, RDE Developers, Enterprise Architecture, Middleware, VC+ Team, Enterprise IT Lab Ops.
Next Update: Thursday, May 23, 10:00 a.m. EDT