Change management using Evolven’s IT Operations Analytics

Learn how Evolven's change management software can help maintain and preserve your environment.

I wrote about techniques for managing change in the data center in early November and referenced some examples of software which can help oversee the process.  One such example was Evolven.  Evolven is designed to track and report change across an array of operating systems, databases, servers, and more to help pinpoint inconsistencies. It can also assist you in preventing issues and determining root causes of problems. Evolven can be helpful with automation—to find out why things didn’t work as expected and what to do next—and can also alert you to suspicious or unauthorized changes in your environment. 

 

Human and technological policies go hand-in-hand to balance each other and ensure the best possible results.  Whereas my last article on the subject referenced the human processes IT departments should follow during change management, I’ll now take a look at technology that can back those processes up by examining what Evolven does and what benefits it can bring.

How does Evolven operate?

Figure A

Evolven works by analyzing the complete software/hardware stack: applications, messaging software, registry keys, files, databases, tables, hardware, virtualization, and Active Directory are among the items and services which it can track. It operates in a traditional client/server mode. A client agent (which can reside on any networked device) collects information about the local configuration and pushes this to a server in the form of an XML file, which is kept in a repository.  The server can be on-premises or cloud-based.  

The initial amount of data might be between 10 and 50 Mb in size. Clients do not transmit this data all at once but in stages so they don’t overload your network (one client won’t use much bandwidth, obviously, but picture the traffic generated by tens of thousands of clients) or internet connection.  As time goes by the clients only send data about the changes made since the initial analysis, which represents just a few kilobytes. Typically the changes are uploaded once per hour.

On that same note, the agent only uses a fraction of the CPU (5% or less) and 128 MB of memory to keep its footprint on the system as minimal as possible. Nevertheless, if this causes a performance impact the agent will exit. You can also schedule the agent only to run at specific times.

Evolven states their product can work as is with 90% of environments and that they can recognize any configuration parameter of a monitored environment, which constitutes hundreds of thousands of unique parameters. You need to set up what’s called an application configuration model for applications built in-house so you can incorporate these into the structure.  

What can Evolven do for me?

Change may be inevitable, but it can be onerous both on a single system and across a group of systems which are supposed to be identical (redundant database servers for instance). Evolven’s job is to measure differences and provide strong analytics to help make sense of these changes—those analytical capabilities are really the key that sets Evolven apart.  

Evolven can compare the current configuration on a system to previous states to see how it matches a "golden baseline." It can also show you anomalies between two cloned servers. It has the ability to display a tremendous amount of data (in meaningful fashion) or something as tiny as one file in a fleet of servers that wasn’t updated successfully.

Evolven is more than change management but also issue avoidance. As a system administrator, I’ve seen some problems take days to fix (and others which never had an identifiable cause, meaning there was still a chance the problem might return). If you can reduce a troubleshooting ordeal from three days to two minutes you’ve saved not only aspirin but operational costs since you can now deal with other tasks.

Here are three real life examples of Evolven at work:

• A company with a medical deployment system has hundreds of servers in production. They had a situation where a critical financial system allowed users to see information held by other user accounts. They used Evolven to find that one of the server files was not updated during a change push. The file was updated and the issue resolved.

• A company deployed new reporting servers from a single clone and suddenly the new systems stopped working. Evolven reported that although the virtual machines were identical the physical hardware underneath was different; there were separate graphical CPUs on the virtual servers. The company contacted the hardware vendor and found the graphical CPU was causing the reporting software to fail.  

• A critical server started to experience performance problems. The company started monitoring and saw I/O increases in the hardware. Evolven pinpointed the fact that one of the developers had turned on logging capability to debug an issue but forgot to turn it off. This caused the server logs to fill up and slowed the system.  

Evolven can send alerts when unwanted changes occur, such as the deletion of a database or an edit to an access control list (ACL). For performance reasons it doesn’t conduct real-time monitoring, meaning you won’t receive an alert immediately after a change takes place, but it can provide warnings within a few minutes after the fact (the scan interval determines the alert interval; this can be set to every 10-15 minutes if required. Evolven staff tells me that most customers find that the one-hour interval meets their needs).  

Extensive reporting capabilities can show you changes in the past 24 hours and beyond, such as software deployments, and you can group these reports based on the severity of the change (critical, uncritical, unclassified and insignificant).

Evolven also allows you to validate that individual changes, patches, and releases are applied accurately and consistently to avoid performance and availability issues and unnecessary stabilization time.

What’s the learning curve like?

Evolven staff says people can get up to speed quickly without extensive technical expertise, pointing out you can customize and implement advanced elements as you get more familiar. I experimented with a trial version of it and was familiar with the navigation within minutes since the graphical interface (see below) is very simple and intuitive.  

The software itself can be set up quickly, with server installations generally taking no more than 30 minutes and clients just 5 minutes or less. Evolven staff stated initial usage in under 2 hours is possible. They have a cloud solution which requires only agents on the client side; no local server component is necessary, which can speed up deployment.

Is it secure?

The security of the product is based on the fact the server has read only access to the clients; the server has no ability to make any changes (also known as remediation) on the clients to address any problems.  

Clients send all their changes to the server using encrypted https connections. You can also set up customized certificates to use for data transmission. Furthermore, the agents utilize authentication credentials when accessing the server and the configuration data can be encrypted. It’s also possible to turn off client transmission and collect local configuration data and transfer it to the server.

Taking it for a drive

I was able to get a first-hand look at Evolven by conducting a demo and experimenting with a trial version of their software.  

Upon first logon the Evolven interface appeared as follows:

Figure B

The tabs across the top represent the main functions: Monitoring, Comparison, Inventory and Administration. They can be described as follows:

• Monitoring: Issue investigation/alerting; for example, checking to see if unauthorized changes have taken place on a system
• Comparison: Comparing one system to another or to itself over time; for example, looking to see if a critical file is different between two servers
• Inventory: Stores information about hosts; for example, view data regarding host configurations
• Administration: Stores Evolven configuration: for example, run reports, set up users, view agent status

In the following screenshot, the Monitoring function shows how a SQL server has been checked to see if anything has changed on it, revealing several database tables have been added:

Figure C

 

You can filter your results by various operational categories to narrow down your analysis. For instance, if you wanted to check and see what might be different about this server in the Performance category, you can select this option:

Figure D

The list of eight changes has been reduced to three, indicating changes to table indexes may be responsible for the performance problems. 

I can further enhance my investigation by performing a consistency analysis, which compares these changes with systems that should be configured the same way. This shows me that only two of the three changes are unique to the problem system:

Figure E

 

See the "Suspicious changes: 1" counter in the upper right? This can help clue you into any unauthorized activity which might be afoot. You can comment on items and flag anything which might be suspicious and isn’t already labeled as such.

Also note the plus sign above All Changes. Clicking the plus brings up the following Breakdown By option to provide more information such as Auto Group, History, Consistency, Authorized vs. Non Authorized and manual selections.

Figure F

Auto Group is a core element to Evolven. This sorts changes and differences into groups to help you with your troubleshooting so you’re not lost amidst a sea of data. Selecting this option shows the following:

Figure G

I can now examine Table Columns individually or I can continue to perform drilldowns. This allows me to efficiently navigate through large amounts of changes.

The Comparison tab can perform different comparison analyses such as "what’s different between Jan 1 and now on these 5 servers" or "compare a golden baseline with these 10 servers." In the following screenshot I can see there are 3,633 differences between a golden baseline and 5 servers, which should be configured the same way.

Figure H

Now, sorting through 3,633 differences sounds exhausting and non-productive. You can filter your results from any time to past 24 hours, past 48 hours and beyond, including custom ranges.  

As shown below, zero changes were made on this system in the past 48 hours which can answer right away the question, "Has anything been altered in the past 2 days, which might be causing the issues I’m seeing with this server now?":

Figure I

Pretty simple, but what if you actually had to navigate through those 3,633 differences? This is where drill-downs, especially Auto Group, comes in handy:

Figure J

In this analysis I was able to drill down by type of environment (SQL Server), impact (critical only), auto group (Tables), and then highlight the source-specific differences. This shows me that two tables were only in my baseline and not in any of my deployed systems.

By drilling down on interesting elements and using the appropriate time-based filters you can zero in on the details that matter.  

The Inventory tab shows you details about your hosts and their environments:

Figure K

It is also where analysis plans (or which environments are compared) can be set up:

Figure L

This can be useful to review any data you might need to find involving your hosts.

The Administration function lets you set up, monitor, and customize Evolven.  For instance, here it shows the Agent status:

Figure M

You can modify the Evolven knowledge base to adjust your categories and set what issues you feel should be critical versus those which are less relevant. For instance, if you feel that the "Element IDX_FIELD_changes_topGuid" item should be considered a Critical item to track in the Performance and Functional groups, you can easily do so:

Figure N

The last item I want to mention is the Reporting function. You can obtain a variety of reports either under the Administration section or within the Monitoring/Comparison sections.

The Administration reports offer the following options:

Figure O  

The Monitoring report options are as shown:

Figure P

 

Here is an example of a report you can run showing the Change summary for your environment:

Figure Q

Accessing Click here to view changes in Evolven shows you the specific details:

Figure R

 

Talking to the creators

I spoke with Sasha Gilenson, the CEO and Founder of Evolven.  Sasha worked at Mercury Interactive, which was in the field of IT management and automation and was acquired by HP in 2006. We chatted about how Evolven came to be:

"I was at Mercury for 13 years, starting as a developer and extending my responsibilities, running the QA department, managing software and service operations and other tasks. Although plenty of change management tools existed at that time, none of them could effectively handle the dynamics and complexity of the data center. The tools didn’t have the granular visibility and analytics capability I thought imperative. The idea came from the experiences we had dealing with IT management and automation.  

"We started working from the application layer, speaking with application and testing teams. The initial idea was to apply the technology of analytics for management of pre-production environments which were dynamic and highly configurable. Our initial discussions were with application folks, QA managers and operation managers. Then we developed the technology and went to our design partners to ask ‘how can we get this to production?’  Virtualization became strong and we added that as another layer, talking to the infrastructure/virtualization folks.

"Once the software was completed, it evolved for the production environment. Evolven was founded in 2007 and it took a couple of years to develop the technology to the point that we released it as a product and then started to sell it. 

"Our target audience is based on the complexity of environment and the criticality of services for organizations. We have large businesses and startups for customers." 

I also spoke with Bill Grant at Evolven, who was kind enough to provide the demo for me, and asked: How does Evolven compare to other configuration/change alerting products such as Puppet and Tripwire?  

Bill replied: "Deployment automation products like Puppet are primarily focused on automating configuration changes. Puppet can also validate they have made these changes correctly, but this is the equivalent of spell-checking your own email; you want the validation using a different lens. Change also happens outside of the deployment process and you want to be mindful of these changes as well. 

Security and Compliance automation products like Tripwire offer security and vulnerability solutions along with an extensive library of compliance and security policies; these are certainly beneficial to CISO’s and corporate security teams. However these solutions aren’t built with IT Operations, Release and Support teams in mind. Evolven’s Analytical approach to monitoring unauthorized changes, validating application and infrastructure releases, and aiding incident investigations is unique in the market and offers direct benefits to these types of users. Evolven can be used independently or in concert with solutions from Puppet and Tripwire."

How much does it cost?

Evolven has two models of pricing; a perpetual model and a subscription model. The perpetual model involves a one-time payment depending on the size of environment. The subscription model offers monitoring/analysis for one server for $30 per month, and there is a discount for more servers.

Completing the analysis

What I appreciate about Evolven is the way it balances simplicity and complexity. The product is easy to introduce to your environment and get up and running immediately, but yet detailed enough that it can provide molecular-level insights into what’s happening on your systems or has occurred over time. The security features it provides ensure that it remains an assistant and not a threat if somehow access to the product is compromised.

I can see several scenarios in my day-to-day role as a system administrator where I could make good use of it – especially those random and sporadic problems that there just isn’t enough time to chase down and put an end to… until they finally become production-impacting. When you calculate the hours spent on company downtime from the perspectives of wasted labor and lost revenue it shows that a detective tool such as Evolven can serve as a healthy insurance policy.

I’d like to thank Sasha Gilenson and Bill Grant for working with me on this review and I look forward to seeing how Evolven continues to develop and grow over time.