What a night yesterday. What a start to my new position and new responsibilities.
We had 2 Connex outages yesterday. One at 9PM, which was a retention issue, and the other was a major interop issue. I could not figure out what’s going on the retention. It looks like a user was prematurely deleted.
The second outage was much worse. It started around 2AM and we were getting pages every 5 mintues. Finally, I woke up and sat around for a while to try to debug this. It turned out that this was an interop failure. As a result, the space was just filling up. Sergey P tried to fix it and I was impressed with his way. But he was not able to fix it. I almost gave up trying… I decided to try one last time… I tried to stop interop, move all log files, and restart. It worked. It took me about an hour and a half. I could not fall asleep since for several hours. 🙂
I am just hoping that this will not repeat. Really. It’s a killer.