<html><head></head><body><div class="yahoo-style-wrap" style="font-family:Helvetica Neue, Helvetica, Arial, sans-serif;font-size:13px;"><div dir="ltr" data-setdir="false">Greetings all,</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">I have what I think is a problem with the ABCL class loader. I am working with a 'big data' library, Spark, but run into an issue on line 2 of the <a href="http://spark.apache.org/docs/latest/rdd-programming-guide.html" class="">programming guide</a> example. I am able to load the JARs from Maven with the ASDF system definition:</div><div dir="ltr" data-setdir="false"><font face=""lucida console", sans-serif"><br></font></div><div dir="ltr" data-setdir="false"><div><font face=""lucida console", sans-serif">(asdf:defsystem #:spark<br> :description "Wrapper for Spark 3.0"<br> :serial t<br> :defsystem-depends-on (abcl-asdf)<br> :depends-on (#:jss #:javaparser)<br> :components ((:mvn "org.apache.spark/spark-core_2.12" :version "3.0.0")<br> (:file "package")<br> (:file "spark")))</font></div><div><br></div></div><div><br></div><div dir="ltr" data-setdir="false">and can create a SparkConf object:</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><div><font face=""lucida console", sans-serif">(defvar *spark-conf*<br> #1"new SparkConf()<br> .setAppName("abcl-app")<br> .setMaster("local")" )</font></div><div><br></div></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">But when I try to create a 'context'</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><font face=""lucida console", sans-serif"><span>(defvar *sc* (new 'JavaSparkContext *spark-conf*))</span></font><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">I get an error in the initialisation:</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><font face=""lucida console", sans-serif"><span>Java exception 'java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.package$'.</span></font><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">There isn't much on this except from the Jenkins guys, who have attempted to put Spark and Spark applications into a CI system. They seem to think that it's related to a call to <a href="https://github.com/apache/spark/blob/a4ca355af8556e8c5948e492ef70ef0b48416dc4/core/src/main/scala/org/apache/spark/package.scala#L59" class="">get a properties file</a> in the package class and on a <a href="https://stackoverflow.com/questions/42751816/spark-version-info-properties-not-found-in-jenkins" class="">StackOverflow discussion</a> suggested that "<span>you should make sure that you set the classloader that Spark was loaded through using the <code>Thread.currentThread().setContextClassLoader(myCustomLoader)</code> call".<br></span></div><div dir="ltr" data-setdir="false"><span><br></span></div><div dir="ltr" data-setdir="false"><span>I've verified with <span>(java:dump-classpath) that the JAR is on the ABCL classpath, and the JAR file does contain the <span><code>spark-version-info.properties</code></span> file. I've also tried getting the file myself with:</span></span></div><div dir="ltr" data-setdir="false"><span><span><br></span></span></div><div dir="ltr" data-setdir="false"><font face=""lucida console", sans-serif"><span><div>(defvar rs<br> #1"Thread.currentThread()<br> .getContextClassLoader()<br> .getResourceAsStream("spark-version-info.properties")" )</div></span></font><div><br></div><div dir="ltr" data-setdir="false">which returns nil<span><span>, so their theory may be correct.</span></span></div></div><div dir="ltr" data-setdir="false"><span><span><br></span></span></div><div dir="ltr" data-setdir="false"><span><span>Messing around with class loaders is a bit beyond my 20 year old Java knowledge so I thought I'd ask here if anyone has any ideas on how I can load Spark in way to use the default Java class loader. Alternatively it occurs to me to ask <i>why </i>the ABCL class loader isn't able to find the properties file if the JAR is on the classpath and then to correct whatever that problem is.</span></span><br></div><div><br></div><div dir="ltr" data-setdir="false">Cheers,</div><div dir="ltr" data-setdir="false"> Steve<br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div></body></html>