In the last several months we’ve all gone through a great deal of change and most of the world has moved into a remote workforce. With this transition into a remote workplace, we’ve experienced tremendous growth over the last few months in response to the changing macro-economic environment. As more and more people are digitizing their workflows and moving to eSignatures, the volume of transactions flowing through our platform has increased dramatically.
We made investments in scaling our systems and thought we were staying ahead of this growth. But, I’m embarrassed to say we’ve experienced 3 outages in addition to brief periods of instability over the last few months. It’s clear that the unprecedented increase in volume of signature requests to accommodate remote work has surfaced short comings in our systems as we’ve had to rapidly scale.
As CEO and someone who’s been working on this company since the beginning, I want to personally apologize for any negative impact our site instability has caused your business. As someone who relies on other vendors to support us as well, I know how painful it is to feel like you’ve been let down. Please know, that we are committed to fixing these issues.
In the interest of transparency, I wanted to touch base to share more information on why this has happened and what we’re doing to make it right.
- What happened: On 9/28, our main database became overloaded due to an influx of traffic.
- What we’re doing about it: We have mitigated this by upgrading our database server and by improving the performance of database queries.
- What happened: Our cloud provider experienced networking issues. This hampered our normal process of scaling in new servers to match increasing load throughout the day. It also caused frequent network errors between our servers and one of the public cloud provider's systems we rely on.
- What we’re doing about it: Our cloud provider has fixed the issue and we are following up with them to understand any further mitigations they are working on.
- What happened: We rolled out some server updates that caused errors. We were alerted to the errors and attempted to rollback. However, a backlog of traffic and tasks caused some of our caching servers to become overloaded.
- What we’re doing about it: We rolled back the bad server update and mitigated the top caching hotspots. Additionally, we are investigating shifting high-traffic cache entries to a separate, isolated cache system and implementing a more robust multi-tier caching system.
For the last 10 years the HelloSign team has been dedicated to putting our customers first. I want to assure you that we are addressing the recent reliability issues with the utmost care and attention. We are reviewing all of our operational processes to make sure we’re strategically building for scale significantly ahead of our current traffic. It’s a top priority for us to deliver the best possible experience to our customers and we are hard at work to make sure we continue to be worthy of your trust.
We appreciate your patience as we work to implement these solutions and are deeply grateful for your commitment to HelloSign.
Co-founder and CEO