Apache Spark Company Product Tech

An Introduction to Apache Spark Optimization in Qubole

An Introduction to Apache Spark Optimization in Qubole

With the growing adoption of massive knowledge for numerous business processes, cloud expenditure is growing more and more extra essential and, in some instances, turning into a limiting factor. As a Qubole Options Architect, I have been serving to clients optimize numerous jobs with great success. In some situations, annual cloud value financial savings ensuing from optimizing a single periodic Spark Software can attain six figures.

Spark is an enormous knowledge software that has been utilized for machine studying, knowledge transformation, and lots of different use instances. Spark supports multiple languages similar to Scala, PySpark, R, and SQL. Spark supplies many configuration parameters that help you optimize the Spark Software. Nevertheless, it will possibly additionally make optimization look intimidating. This text aims to demystify Spark optimization and stroll you through a few of the greatest practices for optimizing Spark within the Qubole setting.

Understanding the Spark Software

Spark consists of a single driver and a number of executors. Spark could be configured to have a single executor, or as many as you might want to course of the appliance. Spark supports autoscaling, and you may configure a minimal and maximum number of executors.

Each executor is a separate Java Virtual Machine (JVM) course of, and you may configure how many CPUs and how much reminiscence might be allocated to every executor. Executors run tasks and allow for distributed processing. A single process can process one knowledge cut up at a time.

Whereas operating your software, Spark creates jobs, levels, and duties. Without going too deep into the mechanics, jobs include levels, and levels include duties. Levels are often executed sequentially, while duties could be executed in parallel within the scope of a single stage.

The Spark UI screenshot under was taken from an precise Spark software. It demonstrates that Spark can run multiple jobs. You may also see quite a few levels and duties executed for every job.

Resource Allocation and Spark Software Parameters

There are 4 main assets: reminiscence, compute (CPU), disk, and network. Reminiscence and compute are by far the costliest. Understanding how a lot compute and memory your software requires is essential for optimization.

You’ll be able to configure how much memory and how many CPUs each executor gets. While the variety of CPUs for each activity is fastened, executor reminiscence is shared between the tasks processed by a single executor.

A number of key parameters provide probably the most impression on how Spark is executed when it comes to assets: spark.executor.reminiscence, spark.executor.cores, spark.activity.cpus, spark.executor.situations, and spark.qubole.max.executors.

To be able to perceive how many tasks every executor will be capable of execute, we need to divide spark.executor.cores by spark.process.cpus. Luckily, except for a number of situations, you need to set spark.process.cpus=1.

Once you realize the variety of tasks a single executor can execute, you’ll be able to calculate the quantity of memory every process can get on average by dividing spark.executor.reminiscence by that number.

Figuring out Resource Necessities

The amount of reminiscence required for a single process is among the most necessary indicators for optimizing the Spark software. This can determine what useful resource (reminiscence or compute) is limiting your specific Spark software and drive your choice on choosing the instance sort. On average, Spark wants between 600MB and 20GB to execute a single process. Nevertheless, there isn’t any common formulation.

One solution to measure this worth is to execute Spark several occasions with totally different configurations, allocating much less reminiscence each time until the job breaks with OutOfMemoryError. Nevertheless, this will turn out to be a really time-consuming course of.

Qubole developed a useful device for Spark optimization – SparkLens. This software offers info on job execution you can make the most of for optimization. One of the values you will discover is PeakExecutionMem (MaxTaskMem in the older SparkLens model). This value represents the maximum amount of memory consumed by a single executor to execute duties throughout actual software execution. This worth is totally different for every stage, and it’s essential to discover the most important worth. Keep in mind that this value does not embrace memory for dataframes, cache, and so forth.

Pay attention to the PeakExecutionMem worth together with the number of duties executed by each executor. PeakExecutionMem divided by spark.executor.cores multiplied by spark.process.cpus will indicate how much memory on average Spark will need to allocate for a single activity to run this specific software. For the instance above, we see that largest executor took 71.1GB. With spark.executor.core=6, a activity will get about 12GB on common.

Another necessary indicator is how many tasks we need to execute in parallel. This info will permit us to correctly measurement the Spark software general in addition to the Spark cluster. Info offered by SparkLens within the Software Timeline section can significantly assist with deciding on the appliance measurement. Pay attention to the taskCount for every stage with a period that’s affordable to deal with. For example, within the screenshot under, regardless that stage 21 processes more tasks, stage 81 takes extra time and is a greater target for optimizing various executors.

Configuring Qubole Spark Cluster

The Spark cluster configuration is nicely described in Qubole documentation out there on-line. We’ll give attention to these cluster parameters which are necessary for Spark software optimization — specifically instance sort, and the minimal and maximum number of autoscaling nodes.

Cloud suppliers supply basic objective, memory optimized, and compute optimized instance varieties. Let’s assessment three AWS occasion varieties in the desk under:


(compute optimized)


(memory optimized)


(common function)

vCPU 16 16 16
Mem, GiB 32 122 64
Value per Hour $zero.68 $1.zero64 $zero.768

Assuming that our Spark software wants 12GB of reminiscence per activity on average, we will calculate how much it’s going to value to run a single process for a full hour on each occasion sort. Word that one vCPU of instance sort usually helps 2 Spark CPUs in Qubole Spark.

c5.4xlarge r4.4xlarge m5.4xlarge
vCPU 16 16 16
Mem, GiB 32 122 64
Value per Hour $zero.68 $1.zero64 $zero.768
# of Duties capacity based mostly on vCPU 32*2/1=64 32*2/1=64 32*2/1=64
# of Duties capacity based mostly on Mem 32/12=2 122/12=10 64/12=5
# of Duties capability based mostly on each 2 10 5
Value of operating a single Process per Hour $zero.34 $0.106 $0.15

As you possibly can see, the fee can differ over 2x relying on the instance sort, and it’s essential to make use of the best instance sort for a selected Spark software.

Configuring Spark Software

Theoretically speaking, a Spark software can full on a single activity. In reality, we want to run a Spark software in a distributed manner and efficiently utilize assets.

Let’s assume that we’re coping with a “commonplace” Spark software that needs one CPU per process (spark.process.cpus=1). We also came upon earlier on this article that our software wants 12GB of Reminiscence per process. We’ll use this value to determine spark.executor.reminiscence and spark.executor.cores. It’s often advisable to configure as giant an executor as potential. Ideally, with r4.4xlarge occasion sort, we should always have the ability to allocate all 122GiB per executor. Nevertheless, OS and different processes additionally need some reminiscence, so spark.executor.reminiscence.overhead is another variable to think about. Its default worth is 10% of spark.executor.memory. With all that, we will create an executor with:

spark.executor.reminiscence = (122-6)*.9=104G

spark.executor.cores = 104GB/12GB=eight

Under is a screenshot from the resource manager for the Spark software with the above configuration.

Now that we’ve selected executor configuration, we will work on determining how many executors we have to efficiently execute this Spark software. SLA and price will usually drive Spark software sizing.

Spark supports autoscaling with these two parameters: spark.executor.situations and spark.qubole.max.executors. Let’s contemplate this instance SparkLens output:

In case you have a really strict SLA and price just isn’t a problem, you would wish to discover a stage with the most important number of taskCount and use this number to find out what number of executors the Spark software will need with a purpose to run all these duties in parallel. Within the screenshot above, the most important taskCount is 14705. With spark.executor.cores=eight, we’ll need 14705/eight=1839 executors. When value just isn’t a problem, we will set each spark.executor.situations and spark.qubole.max.executors to 1839. The cluster also needs to be configured properly to help this Spark software.

Sadly, in real life we have now to handle value while attaining SLA. As a way to obtain that stability, we need to make sure that the compute and memory assets are utilized in probably the most efficient method. In the screenshot above, stage 21 takes 1m 44s to course of 14705 tasks and stage 81 takes 4m 55s to course of 2001 duties. Optimizing for stage 21, we’ll want 1839 executors. Optimizing for stage 81, we’ll need 251 executors. It’s clear that optimizing for stage 81 will save us near 90% of the fee without vital influence on efficiency!

The Spark software might be further improved with autoscaling. Concentrating on 1839 executors for all the software run, more often than not these executors will idle as there will not be sufficient tasks in a lot of the levels to maintain them busy. Having solely 251 executors will slow down some levels. To resolve this dilemma, we will set spark.executor.situations=251 and spark.qubole.max.executors=1839 and that may permit Spark to scale the appliance out and down relying on the needs of varied software levels. Autoscaling permits for probably the most efficient use of assets and works greatest on a Spark cluster shared amongst many purposes and users.

Revisiting Spark Cluster Configuration

Empowered with values for spark.executor.situations and spark.qubole.max.executors, we will derive the minimum and maximum number of autoscaling nodes. Needless to say clusters are often shared assets and can normally run multiple jobs for a number of customers at a time. The autoscaling nodes shall be shared between all concurrent jobs.

Validating Configuration

Now for the exciting part! Let’s run the Spark software with the brand new configuration and validate our assumptions.

First, validate that nodes are maxed out on either compute or reminiscence. The nodes page of the resource manager is an effective software for that. For instance, the screenshot under indicates that about 4GB of memory can still be utilized.

Second, evaluate SparkLens output to verify what number of executors have been operating on every stage and give attention to the levels that we goal with our optimization.

Third, and foremost, did we achieve the SLA? Is there room for more optimization? Optimizing Spark software is an iterative process and may take time. Nevertheless, the fee benefit that may be achieved is often properly well worth the efforts.

Different Issues

A driver will use one CPU on a single node. If executors are configured to utilize all CPUs usually obtainable on an occasion, the node operating the driving force will be unable to run a single executor as it is going to be one CPU brief.

Knowledge format might have a big impression on Spark software performance. Knowledge is often skewed, and knowledge splits aren’t even. Spark assigns knowledge splits to executors randomly, and each Spark software run is exclusive from that perspective. Within the worst-case state of affairs, Spark will place all giant knowledge splits to a single executor at the similar time. Whereas that is an unlikely state of affairs, the appliance configuration may have to accommodate for it. Nevertheless, keep in mind that Spark will try to re-run failed duties.

Knowledge being processed by the Spark Software can possible change, and a tightly optimized Spark software might fail when knowledge is floating. You want to contemplate the complete end-to-end process when optimizing Spark software.

Some jobs and levels could possibly run in parallel. SparkLens stories such instances. In that case, think about levels operating in parallel as a single stage from an optimization perspective. SparkLens stories on the important path. Pay shut consideration to that output, as it might considerably help with concentrating on specific levels for optimization.

Be cautious when optimizing a Spark software that has many levels that process many tasks but take lower than a minute or two to course of. Autoscaling takes time and should not have the ability to autoscale quick enough. See Qubole’s Spark autoscaling document for more info.

The code (Scala, Python, and so on.) that Spark software runs might be also optimized. While this is not in the scope of this article, I recommend taking a look at broadcasting, repartition, and other means of optimizing your code.

Spot situations on AWS is one other powerful value administration device that Qubole supports very effectively.

Parallelism for giant jobs could also be limited by default worth for spark.sql.shuffle.partitions. Contemplate growing this from default 200.

Control spills. Small spills might be an indicator of a stage operating tight on reminiscence. You possibly can see spills in the SparkLens and within the Spark UI.

Lastly, occasion sort with high I/O comparable to i3 might considerably improve efficiency of Spark software operating plenty of knowledge shuffling.

In Conclusion

This article is just an introduction to Spark software optimization. You’ll find out extra about Spark tuning in our tech speak replay from Knowledge Platforms 2018 about Supercharging the Efficiency of Spark Purposes.

Qubole also offers Spark optimization as a part of our Skilled Providers, so be happy to reach out to debate your wants.