An ITSM Key System Upgrade
Project Summary
Our Client, a major Global Metals trading company, was running an outdated Jira Service Desk version which was creating a downstream impact to the operations of their technical support team. The core software was multiple major versions behind and close to two years old, lacking several key new features that Atlassian had introduced in the most recent upgrades. Most importantly, the existing JIRA Service Desk was missing major security patches which made the upgrade process critical and urgent for our Client. It was important to minimize any security risk and handle the upgrade to close any vulnerabilities as fast and efficiently as possible.
Constraints
- Extremely tight schedule
- Limited possibility for a downtime
- No mirror environment on Client’s premises
- VMware as the only hosting solution
- Late hardware delivery due to Covid-19 situation
Locations
Project delivered from: Sofia, Bulgaria
GEO Scope: Global
Due to the well prepared cookbook it was possible even for a junior technical personnel to execute the actual upgrade – following our guide step-by-step with the actual steps and commands.
The Solution
We worked closely with the customer to identify the best possible implementation plan, having in mind that any downtime to the production environment had to be kept to a minim. At the same time, we had been specifically requested to factor in a thorough testing phase by the Service Desk Team management to minimize the risk of having any issue after go-live. To make the case even more challenging, creating a mirror environment on customer premises was not possible at the time due to a concurrent hardware replacement on the Testing site – Our Client was in the middle of a hardware upgrade, being their test environments close to 100% utilization. Waiting for the new test environment to be up and running would have meant a delay of 4 to 6 weeks to a minimum and we were asked to look for alternative options.
We created a project plan that took into account the priority given to closing system vulnerabilities while not having access to the server due to planned system down and as result we decided that the best approach was to build an identical environment on our premises, test the upgrade, verify that everything worked as required and finally hand over to the customer for full testing. Two more obvious challenges occurred – we did not have VMware as a virtualization option on our testing environments on one hand, and how we can be as close aspossible to the customer’s environment, in order to guarantee that the solution will be exactly the same on the other hand. We discussed with the customer and found a solution that satisfied all parties – the customer would clone and obfuscate the production environment and would provide us with a VM export from the Production VMware virtualization server.
We received an encrypted laptop, holding a VM with the obfuscated Jira Service Desk environment. The next challenge was the fact that we did not have spare server at that time to run VMware virtualization on. One option was to convert the received VMware image to a Proxmox (the virtualization we were running on the available hardware) image and do the upgrade there. This, however introduced multiple additional steps, each of which could’ve contaminate the whole process and would’ve moved the task a little bit away from the customer’s solution. We decided to create a VM inside the Proxmox server and install VMware as a nested virtualization – we’ve had previous positive experience with similar situation and decided it would be a good solution also in this case.
Spinning up the VM was really a straight-on task and with the proper configuration, everything was working perfectly. We then imported the VM into the VMware machine and started the actual upgrade process. During the first iteration, we hit few bumps along the way due to some specifics the customer environment had. The positive side about running everything on a virtual machine, is the easy process of reverting all changes back – we’ve made a snapshot of the recently imported VM and we were able to do few iterations of the upgrade process until we’ve had the whole process tested, verified and documented. The last two iterations were just to confirm that the upgrade cookbook created over the testing, were perfect – and perfect they were – following the cookbook step-by-step showed a perfect result – no errors whatsoever during the process. We decided that it was time to provide the VM back to the customer in order for them to test the upgraded Jira Service Desk but due to the COVID-19 situation, their hardware delivery was delayed with a couple of weeks, which would’ve had negative impact to the whole upgrade timelines. We decided instead to create a custom VPN solution in an isolated network so that the customer could test on our hardware while they waited for theirs. It took two whole weeks of testing from customer’s side and everything looked good. In the meantime they had received and set up their testing equipment so we decided that it was now time for one last test – we asked the customer to clone the Prod environment on their Test environment which was done successfully on the same day.
Afterwards we went once again and followed the upgrade cookbook – the outcome was an easy and straightforward process. Quick test from the customer confirmed that everything was working as expected. One last step was left – to perform the actual production upgrade. We discussed a number of options with the customer and after agreeing on the best date and time, we had green light to start. As expected the upgrade went flawlessly due to the really good preparation and multiple testing. Due to the well prepared cookbook it was possible even for a junior technical personnel to execute the actual upgrade – following the guide step-by-step with the actual steps and commands. The downtime of the production environment was actually less than an hour and all services were up and running on the 45th minute of the upgrade start. The customer tested and accepted the upgraded Jira service desk without any problems.
The Result
We managed to move fast, reacting to a specific need of our Client – a real life vulnerability risk – by upgrading a key system, hitting a tight deadline while managing a number of challenges on the way. The Client was very pleased of the fact that they had been given ample time for testing while keeping to a minimum the amount of downtime of the production environment. Their software environment is now incorporating the latest features and followed the best in class security practices, as it was also verified by the penetration testing that was run following the upgrade.
Synergyc showed that it is a partner that can be relied upon when response time, thorough planning and laser focus on execution are required. Flexibility and adaptability are also in our DNA, ensuring that all risks and last minute challenges are dealt with and mitigated swiftly as part of our standard ways of working.