Correct Way to Add Library Dependency in build.sbt in IntelliJ

In this post we will look at the correct way to add library dependency in build.sbt in IntelliJ.

Open the "sbt.build" file.

Taking the example of adding the Spark Core library dependency, add a "libraryDependencies" line of code as shown below.

Quick Tip 1: For this post I am going to choose the latest point version (2.11.x) of Scala version that is mentioned in the official Spark website, in order to maintain compatibility. Always check for which Scala version to use with Spark before starting your journey.

Quick Tip 2: The double percentage (%%) in the librarydependencies code tells the IDE to download the Spark Core libraries corresponding to the Scala version mentioned in the build.sbt file. In the below case the IDE will find and automatically download the Spark Core libraries that were built using Scala version 2.13.3.

Initially, right after adding the dependency, if you hover the mouse over the line of code, the IDE will throw a message saying - "Unknown artifact..."

In some cases, if auto import is enabled in the IDE, then right after add the libraryDependencies line of code and saving the file, the process of downloading the dependency should start.

However, in many cases it may not work automatically (as in my case). In order to resolve this, click on the "Load sbt Changes" button in the top-right corner of the code editor, as shown below.

After this, IntelliJ will take a while and start loading the Spark Code libraries. If it is the first time, that this library is being downloaded to your computer and if you internet speed is slow, then be prepared to wait even for an hour or so. However, in most cases, nowadays the process is faster - thanks of fast internet of the modern era! In my case it took about 7-8 minutes. A lot of verbose logs would appear. Just sit tight through the process.

In the end, once the download is successful, you will notice that the new libraries are listed under the External Libraries node.

Word of encouragement: In case if IntelliJ hangs, though that may not be often, don't give up. After a few tries this should work.

That's all! You're all set!