Fine-tuning resource allocation of apache spark distributed multinode cluster for faster processing of network-trace data

Shyamasundar, LB and Anil Kumar, V and Jhansi Rani, P (2019) Fine-tuning resource allocation of apache spark distributed multinode cluster for faster processing of network-trace data. International Journal of Advanced Computer Science and Applications, 10 (11). pp. 634-645. ISSN 2158107X

Text Paper_84-Fine_Tuning_Resource_Allocation.pdf Restricted to Registered users only Download (3MB)

Abstract

In the field of network security, the task of processing and analyzing huge amount of Packet CAPture (PCAP) data is of utmost importance for developing and monitoring the behavior of networks, having an intrusion detection and prevention system, firewall etc. In recent times, Apache Spark in combination with Hadoop Yet-Another-Resource-Negotiator (YARN) is evolving as a generic Big Data processing platform. While processing raw network packets, timely inference of network security is a primitive requirement. However, to the best of our knowledge, no prior work has focused on systematic study on fine-tuning the resources, scalability and performance of distributed Apache Spark cluster (while processing PCAP data). For obtaining best performance, various cluster parameters like number of cluster nodes, number of cores utilized from each node, total number of executors run in the cluster, amount of main-memory used from each node, executor memory overhead allotted for each node to handle garbage collection issue, etc., have been finetuned, which is the focus of the proposed work. Through the proposed strategy, we could analyze 85GB of data (provided by CSIR Fourth Paradigm Institute) in just 78 seconds, using 32 node (256 cores) Spark cluster. This would otherwise take around 30 minutes in traditional processing systems.

Item Type:	Article
Uncontrolled Keywords:	Big data; packet data analysis; network security; distributed apache spark cluster; Yet Another Resource Negotiator (YARN); parameter tuning
Subjects:	MATHEMATICAL AND COMPUTER SCIENCES > Mathematical and Computer Scienes(General) MATHEMATICAL AND COMPUTER SCIENCES > Computer Operations and Hardware MATHEMATICAL AND COMPUTER SCIENCES > Computer Programming and Software
Depositing User:	Smt Bhagya Rekha KA
Date Deposited:	06 Jun 2022 15:41
Last Modified:	06 Jun 2022 15:41
URI:	http://nal-ir.nal.res.in/id/eprint/13605

Actions (login required)

View Item