The NICE Nexidia DevOps team uses the SaltStack automation platform to create consistency and control across 100+ single-tenant Windows environments—each with its own bespoke customer requirements.
NICE Nexidia performs maintenance and server updates every six weeks. Since the environments are Windows-based, the team must reboot all 900 or so servers across all 100 single-tenant environments—sometimes multiple times. Prior to SaltStack, this was a massive undertaking that required extensive pre-planning to account for data restrictions, exceptions, and regional seasonality (such as holidays and tax seasons).
The team is required to adhere to a strict 48-hour maintenance window and the update process typically involved 24 engineers working more than 30 consecutive hours to ensure each server, process, and application was taken offline, updated and brought back online in the correct order to avoid data loss or downstream connectivity issues. Once the updates were complete, the team would need to manually log in to each environment, troubleshoot issues, and ensure the environment was working properly before the customer resumed operations on monday morning.
By replacing their manual update process with SaltStack event-driven orchestration, NICE Nexidia reduced the 30-hour consecutive effort down to 15 hours broken up across two days. In addition, the operation that used to require 24 engineers can now be managed and tested by a team of five, an 80% reduction in work hours. SaltStack has also greatly increased the consistency and reliably of the process. SaltStack uses intelligent orchestration to restart each service in the correct order, ensuring that the system comes back up in a good working state.