The Beginning of the End

Tomorrow might be my last day in the current project. Though I have heard these words for the last 6 months, the end seems to be definite this time. The last 19 months of staring at the same status screen and recovering the same loads again and again have taught me a lot that includes both technical and managerial aspects of the project. I even learnt tips of tricks of handling the politics that run within the IT industry especially in the client space when you are not the most favorite to them and survival depends more on your smartness than hard work in creating the solution. Between the time of learning the code which God knows who developed and even He might not know who maintained, you will have to use your social skills in getting information from your co-workers who see you as a threat to their job. So it’s a tough task in getting the job done and also getting the credit for it. But there are also few good souls who would recognize your efforts in a while and you are motivated to work for those appreciation mails. This Project had not made me work longer weekends than my previous ones but this indeed made me face tougher challenges and sleepless nights as while you prepare yourself to play the blame game with other teams in the project, your offshore sometimes trip you like a lousy sidekick by pressing the wrong buttons out of sheer curiosity.
Though there are many incidents that I might be forced to create a new blog titled "My pain and the gain with the Cable Guy", I will put up a few events that are at the top of my memories.

The Beginning of the Never-ending story
Brought to onsite for created GPS reports (Please watch Wanted movie), before I could set my hands on them properly, my PM called me for a meeting. These are the exact words he said.
"Our project is in a very bad state. The client might throw us out if we do not do something about it. And it is all in your hands. I am putting you in charge of Production Support team and you must bring the team up and create a good name with the client. You can do it."
Everyone who would hear such inspiring words from your manager and assuring that the world rests on your hands; it probably would have created enlightenment in your mind and brought Goosebumps inside you. But for me, it tickled and I laughed out loud for I heard him saying the same dialogue to S to persuade him join another module. We all love our managers for their convincing words. Don’t we? But I hoped he could have changed the words at the least.

The night I almost screwed up
Not even a month of leading the Production support from onsite, I had a simple instruction to run specific jobs in the night. And that day happened to be a party night and everyone were at home and we were enjoying a good dinner. The clock struck 11 and I went on to my start the specified jobs only to find that a variation of that job is already active from 10. The schedule for the new jobs was at 10 and not at 11 which I completely forgot. Not willing to call the client to face embarrassment, I took the bold step of going through the entire code, understand it, find the pros and cons of running it after 11 and created the detailed steps to complete it successfully. The plan went on for the whole night and it took 20 hours to finish the whole load. But still I had to spend another sleepless night as I was not sure if I made any mistake that the client might notice the next day. But fortunately, everything went fine and nobody even suspected the manual overrides done on the weekend. It taught me a huge lesson. Always remember that you CAN do it, if you WISH for it and WORK HARD to make it happen. And also make sure you note down the time in your schedule correctly.

24 hours conference call and $25 movie tickets
Almost an year back during the labor day weekend, when everyone in the office were on their long planned vacations while I was just sitting in home with no plans, the server crashed and forced all of them to join the conference call. With everyone's vacation ruined and me with nothing to be ruined, we spent 24 hours in the call while engineers from various team tried hard in bringing back the crashed server to life. As the issue was not directly on my application, I had nothing to do in the whole time except keep the mobile charged and was left lying on the bed for the whole day. Though my brain tried to follow up the progress of the issue, my mind could not concentrate on the call owing to the sleepiness night. My only real work was running the test loads on the next day and declaring that all things are fine. And the reward: A signed appreciation letter (hard copy) from the client along with $25 worth movie tickets. Dedication pays off sometime with style.

The night I thought I screwed up
This happened few months back when we saw failure on a module which never had any problem before. With preliminary analysis done and the solution derived to solve the problem, we proceed with applying the fix. As the normal process, we are supposed to take backup of the data before doing any changes in the system. Even though I am always bit of extra precautious in handling the production systems and would rather be a road block than a open canal for care-free developers, on this particular day I was partially out of my mind and allowed the code execution on the production without any backups. As if it is a punishment for my carelessness, the tests failed and were showing weird output in the test environment. As I had proceeded with the production before verifying the results from the test environment, I knew I am in deep trouble. The whole weekend became a nightmare with all strange thoughts from being branded as the reckless guy in the project to getting fired by the client flashing in my dreams. Hard it was to understand that what was done is already done and there is nothing you can do to undo it. As we did not even take the backup, so recovery from an old backup is a costly option which was closer to impossible. With alarms running through my mind, it was Hemant's unimportant question whether I had run the tests on the Production environment itself. And that was when it struck me that we never ran complete tests in production and were presuming by looking at the similar test environment. A quick test finally revealed to my surprise that the production does not have any issues and the test environment was already corrupted before we had applied our fix. Breathing at last, I laughed at myself for making judgment calls without complete information. Another important lesson learnt from the day. Facts have better answers than your assumptions. It is never too late to recheck your results. Sometimes it is advisable to listen to your colleagues.