[SPARK-56696][BUILD] Exclude old junit:junit from the dependency tree for sbt build

### What changes were proposed in this pull request?
This PR excludes old `junit:junit` from the dependency tree for sbt build.
This is also another solution for SPARK-56478 (#55358).

### Why are the changes needed?
Currently, we use `org.junit.jupiter:junit-jupiter` rather than `junit:junit`, and `junit:junit` is explicitly excluded in `pom.xml`.
https://github.com/apache/spark/blob/8e37824402531f82ad1dfef415b5c29b478df760/pom.xml#L1519

But the dependency tree for sbt build still includes `junit:junit`.

```
$ build/sbt Test/dependencyTree

...

[info]   | +-org.apache.parquet:parquet-column:1.17.0
[info]   | | +-com.carrotsearch:junit-benchmarks:0.7.2
[info]   | | +-junit:junit:4.13.2
[info]   | | | +-org.hamcrest:hamcrest-core:1.3
[info]   | | |
[info]   | | +-org.apache.commons:commons-lang3:3.17.0 (evicted by: 3.20.0)
[info]   | | +-org.apache.commons:commons-lang3:3.20.0
[info]   | | +-org.apache.parquet:parquet-common:1.17.0
[info]   | | | +-junit:junit:4.13.2
[info]   | | | | +-org.hamcrest:hamcrest-core:1.3
```

`junit:junit` also depends on `org.hamcrest:hamcrest-core:1.3`, which is causes an issue. If, we build spark using Maven with `-Psparkr`, the pom file for `hamcrest-core:1.3` will be downloaded but the corresponding jar file will not.
```
$ build/mvn -DskipTests -Psparkr package
$ ls ~/.m2/repository/org/hamcrest/hamcrest-core/1.3/
_remote.repositories  hamcrest-core-1.3.pom  hamcrest-core-1.3.pom.lastUpdated  hamcrest-core-1.3.pom.sha1
```

If we then build Spark using sbt in this situation, it will fail.
```
$ build/sbt package

...
$ build/sbt package

...

[error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] file:/home/release/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar: not found: /home/release/.
m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar
```

For the same reason, building Scala/Java API document will fail.
```
$ cd docs
$ SKIP_RDOC=1 SKIP_PYTHONDOC=1 SKIP_SQLDOC=1 bundler exec jekyll build

...

[error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] file:/home/release/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar: not found: /home/release/.
m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar
```

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Confirmed both of the following commands succeeded
```
$ build/mvn -DskipTests -Psparkr package
$ build/sbt package
$ cd docs && SKIP_RDOC=1 SKIP_PYTHONDOC=1 SKIP_SQLDOC=1 bundler exec jekyll build
```

### Was this patch authored or co-authored using generative AI tooling?
No.

Closes #55645 from sarutak/exclude-old-junit.

Authored-by: Kousuke Saruta <sarutak@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 74ac7f4)
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

Author: sarutak

docs/_plugins/build_api_docs.rb

@@ -45,10 +45,6 @@ def build_spark_if_necessary
 
   print_header "Building Spark."
   cd(SPARK_PROJECT_ROOT)
-  # Maven may leave POM-only org.hamcrest:hamcrest-core trees under ~/.m2; SBT/Coursier then
-  # fails with "file:.../hamcrest-core-*.jar: not found". Clear before invoking SBT.
-  hamcrest_m2 = File.join(Dir.home, '.m2/repository/org/hamcrest/hamcrest-core')
-  FileUtils.rm_rf(hamcrest_m2)
   command = "NO_PROVIDED_SPARK_JARS=0 build/sbt -Phive -Pkinesis-asl clean package"
   puts "Running '#{command}'; this may take a few minutes..."
   system(command) || raise("Failed to build Spark")

project/SparkBuild.scala

@@ -1288,7 +1288,8 @@ object ExcludedDependencies {
       ExclusionRule("org.slf4j", "slf4j-simple"),
       ExclusionRule("javax.servlet", "javax.servlet-api"),
       ExclusionRule("io.netty", "netty-codec-protobuf"),
-      ExclusionRule("io.netty", "netty-codec-marshalling"))
+      ExclusionRule("io.netty", "netty-codec-marshalling"),
+      ExclusionRule("junit", "junit"))
   )
 }