Parallelizing FlowView: Performance Improvements

by Debojit Dhar on August 05, 2014

FlowView provides detailed insight into the applications on one’s network, but to achieve this, we need to store a lot of data. Behind the scenes, when we are trying to get top applications or top hosts and categories, FlowView has two available data sources: Google BigQuery and raw capture files. BigQuery is fast for many use cases, especially when the data set is large and does not have to be real-time. However, up-to-the-minute data analysis over moderately large datasets, we parse the raw files directly to extract the information we are looking for. Sometimes, when looking up data distribution for long intervals or prying into appliances which are generating significantly large amounts of traffic, the response time in our FlowView UI is a point of concern.

To tackle this issue, we introduced parallel processing of flow files as part of our release last month. Essentially, any information we see on our FlowView screen translates to an nfdump command that reads through the data uploaded from individual appliances. Further, all requests from our FlowView UI are time bound - they show data within a start and end time interval. This, in turn determines the number of flow files that need to be processed by the FlowView backend. In our previous implementation, each FlowView request would be served by a single thread which would gain access to the resources required to serve the request. The time taken to serve each request would is generally governed by two major factors - the time range requested and the flows per second (fps) of the appliance.

Shown below are the sample appliances we used in this experiment.

Appliance/GUID

Model

Interface

Flows-per-second

No. of nfcapd files

Time Period

Size on disk

We also chose a few frequently requested operations that we wanted to optimize.

• Traffic Summary
• Top Applications
• Top Hosts
• Host Details

Shown below are the initial response times from these operations

Traffic summary

Note that higher rate appliances do not all 30 days of data as the file size exceeds the limit of nfdump and need to be fetched from Big query where this optimization does not apply.

Top Applications

Top Hosts

Host Details

As evident -

• when querying an appliance emitting more flow records per second, the response time is generally slower.
• when querying over larger time intervals, response time is slower.

The response time corresponds to the number of files that need to be processed and of course the size of individual files.

To counter this issue we introduced a logic that splits individual FlowView requests into multiple execution threads and run them in parallel. The total time period requested is divided into equal chunks based and the results from each chunk amalgamated. Now the first question to be asked was when to split? and how many threads to split into? Every request that is made had different performance gain/loss when split into threads and based on this knowledge - an algorithm needed to be introduced.

Our approach to solve this was purely heuristic - get a framework in place that would support splitting and merging threads, start from a low number and scale to larger number of threads, measure performance gains across appliance types over different intervals.

The results from this experiment is shown below :

Note that time required to query moderate rate appliances is sometimes close to or even longer than the corresponding high rate appliance. The charts show data from each type of appliance but when calculating gain - only data from a particular appliance is compared.

As evident from the results, for each of the four different queries, appliances with large flow rates initially had slower query times and show the best improvement results for queries ranging from short to long intervals. While appliances with slower flow rates only show improvement when the query interval is significantly longer. This led to a simple logic to be in place. The appliances with higher flow rates are generally rack appliances (r40 and r400) - would benefit most from splitting into 4 threads when requests encapsulate data for anything more than an hour. The lower flow-rate micro appliances (m20, m22 and m30) would only benefit from thread splitting when time range for data requested is over 24 hours and two threads are sufficient. For larger number of threads, the overhead of splitting and joining overweighs the performance gains achieved from parallel execution. As we increase the interval to about a month, even appliances with low flow rates start to benefit from having 4 threads or even more.

To keep the design simple and conservative, we started with 2 threads for micro appliances for requests over 24 hours and 4 threads for rack appliances for request over 1 hour. Introducing this improvement has shown great improvements in the flowView UI - some queries which were almost unusable before now show significantly improved response times. This method demonstrates how simple improvements can go a long way in improving performance. The tactics are so generic that it can be used in any system suffering from performance issues when querying large number of data chunks.

FlowView users should have a better experience since this release has been pushed to our servers and stay tuned for more improvements to come in the future.