At this time all services have been brought back online. We are still monitoring systems for stability and to ensure continued uptime.
Posted Apr 16, 2022 - 22:35 EDT
Update
We are still working with our 3rd party vendor and will have more updates in 48 hours. At this time data integrity is our highest priority.
Posted Apr 15, 2022 - 22:57 EDT
Update
Dear customer,
You are our priority, and we know for those that are supporting internal and external customers, our recent challenges have been distressing for you.
We have been working with our 3rd party partners and have developed a plan to achieve stability. Regrettably, this will impact a portion of our customers, but it is crucial to achieving a sustainable solution. If you cannot access our services at this time, you should plan for a 40 hour service outage. We have insulated our customers who can access our services from outages during this period.
I understand that stating our most profound regret doesn’t erase the impact you have experienced. My team is doing everything in our effort to resolve this and come out on the other side better for it. Following the guidance from our 3rd party partners, we are making substantial financial investments in our environment that will provide a more robust and stable solution for the foreseeable future.
Sincerely, John H. Hurley CEO/Co-Founder
Posted Apr 14, 2022 - 12:01 EDT
Identified
We continue to work through issues associated with our newer database solution. Those customers that are migrated to this have been disabled. We are currently engaged with our 3rd party support team reviewing our findings from this morning’s events. The next update will be at 11 am eastern,
Posted Apr 14, 2022 - 10:06 EDT
Update
We have been able to bring more customers back online. There is still a small group of customers we are working on.
Posted Apr 13, 2022 - 12:37 EDT
Update
There are still a group customers with issues at this time please stand by for more updates to come in around 3 hours.
Posted Apr 13, 2022 - 09:24 EDT
Monitoring
New hardware components have been put in place and services are operation with some limitation at this time.
Posted Apr 12, 2022 - 22:47 EDT
Update
We are continuing to work on a fix for this issue.
Posted Apr 12, 2022 - 19:55 EDT
Identified
We experienced a reoccurrence of the metadata corruption issue that occurred this past weekend. Before taking any recovery steps, we engaged our 3rd party vendor.
Working with them, we identified the root cause and resolved it without the 30-hour metadata recovery.
To minimize the risk of additional service outages, we are taking an emergency outage from 9-11 PM eastern this evening, April 12, 2022, to upgrade a critical hardware component.
Posted Apr 12, 2022 - 13:20 EDT
Update
We are seeing some service degradation on select site and working on them.
Posted Apr 12, 2022 - 13:14 EDT
Monitoring
All services have been restored.
Posted Apr 11, 2022 - 23:00 EDT
Update
The first phase of metadata recover have been computed successfully. We have now started the 2nd phase. Some customers sites are still inaccessible at this time and our share link service is still down. Our next update will come at 9:30am Eastern
Posted Apr 11, 2022 - 19:46 EDT
Update
As stated in the CEO communication, we have customers still using the original database technology and some customers using the newer one. We have developed a solution that allows access for those customers on the original database technology.
If you cannot access your data at this time, please note you are on the newer technology, and your metadata is still in the process of being rebuilt. We still do not have a high confidence level on the completion time; however, we know that this is a 2 phase process and phase 1 has been running now for 20 hours. We can reasonably expect that phase 2 will take equally as long.
We’re sorry for the major impact this has had on you as we know our service is critical to your business.
Posted Apr 11, 2022 - 12:48 EDT
Update
We continue to monitor the recovery process. We will have our next update around 6pm EDT unless there is any new developments.
Posted Apr 11, 2022 - 10:07 EDT
Update
From the CEO Dear Customers,
I am writing this to provide you with additional information about the service outage we are currently experiencing. At the onset of the outage, the team reported a hardware failure. We determined that the cause is due to work we now have in progress improving our technology. This improvement is necessary to ensure we are well-positioned for the future.
For months, we have been migrating customer data from a database technology that we have been using for many years to a more robust solution. The newer technology experienced an issue with its metadata. After engaging our European 3rd party vendor, considered experts with this technology, we have determined that the metadata pool that became corrupted will require to be rebuilt. The software manages this process. Unfortunately, this requires us to prevent new updates from occurring until it has been completed. This affects all of our customers, both on the latest and legacy databases.
The process does not provide a method for us to estimate when the rebuild process will be complete, but we are still looking for ways to provide us with some level of estimating.
All customer data is intact. This issue is limited to metadata. I want to emphasize that no customer data has been compromised.
We know that you have trusted us with your data and, in some cases, your customer’s data. We take that responsibility very seriously, which is our motivation for investing in this newer technology.
Without a tool to estimate the completion, we are providing a best-case recovery of noon eastern. If we can acquire a more specific estimate before this time, we will give an update, but at a minimum, update the status at noon.
John Hurley CEO/Co-Founder Lumanox, LLC
Posted Apr 10, 2022 - 22:41 EDT
Update
We are still working on recovering services from a major hardware failure.
Posted Apr 10, 2022 - 14:06 EDT
Identified
The issue has been identified and a fix is being implemented.