<html><head></head><body><div class="ydpba2a52b4yahoo-style-wrap" style="font-family:Helvetica Neue, Helvetica, Arial, sans-serif;font-size:13px;"><div></div>
<div dir="ltr" data-setdir="false">Thank you Olof; that was just what I needed to get things working. Well, that and another half day struggling with what turns out to be a 5 year old bug (<a href="https://abcl.org/trac/ticket/338">#388</a>). I <b>really </b>wish someone had mentioned in the JSS documentation, "Oh, and this doesn't yet work in top-level forms". All told, it was a lot more difficult to get started with ABCL than I expected it to be, but I'm glad it's done and grateful to those that helped. For reference, here's the code that finally works:</div><div dir="ltr" data-setdir="false"><font face="lucida console, sans-serif"><br></font></div><div dir="ltr" data-setdir="false"><div><font face="lucida console, sans-serif">(defun change-class-loader ()<br> (#"setContextClassLoader" #1"Thread.currentThread()" (java:get-current-classloader)))<br>(change-class-loader)<br><br>(defun make-spark-config (&key (app-name "abcl-app") (conf-master "local"))<br> "Return a spark configuration.<br>Required to work around ABCL bug 388, otherwise we'd just do this at a<br>top-level form. See https://abcl.org/trac/ticket/338"<br> (let ((conf (jss:new (jss:find-java-class "org.apache.spark.sparkConf"))))<br> (java:chain conf<br> ("setAppName" app-name)<br> ("setMaster" conf-master))))<br><br>(defun make-spark-context (spark-config)<br> (jss:new 'JavaSparkContext spark-config))<br><br>;;; Now we can create our context and configuration object<br>(defvar *spark-conf* (make-spark-config))<br>(defvar *sc* (make-spark-context *spark-conf*))</font></div><div><br></div></div><div dir="ltr" data-setdir="false">At least it gets me as far as line two of the spark 'hello world'; hopefully there aren't any other surprises lurking. If anyone can recommend any best practices or improvements, especially around the class loader bits, I'd be very happy to hear them.<br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Regards,</div><div dir="ltr" data-setdir="false"> Steve<br></div>
</div><div id="yahoo_quoted_6043243240" class="yahoo_quoted">
<div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">
<div>
On Wednesday, July 22, 2020, 6:59:32 PM GMT+8, Olof-Joachim Frahm <olof@macrolet.net> wrote:
</div>
<div><br></div>
<div><br></div>
<div>On Wed, Jul 22, 2020 at 09:52:45AM +0000, Steven Nunez wrote:<div class="yqt6600991273" id="yqtfd40762"><br clear="none">> I've verified with (java:dump-classpath) that the JAR is on the ABCL<br clear="none">> classpath, and the JAR file does contain the<br clear="none">> spark-version-info.properties file. I've also tried getting the file<br clear="none">> myself with:<br clear="none">> (defvar rs<br clear="none">> #1"Thread.currentThread()<br clear="none">> .getContextClassLoader()<br clear="none">> .getResourceAsStream("spark-version-info.properties")" )<br clear="none">> which returns nil, so their theory may be correct.<br clear="none">> Messing around with class loaders is a bit beyond my 20 year old Java knowledge [...]</div><br clear="none"><br clear="none">Just to get you a bit unblocked, it seems you can indeed set the current<br clear="none">context class loader and then the call to create the `JavaSparkContext`<br clear="none">succeeds:<br clear="none"><br clear="none">```<br clear="none"># verify that it doesn't work by default<br clear="none">CL-USER> #1"Thread.currentThread().getContextClassLoader().getResourceAsStream("spark-version-info.properties")"<br clear="none">NIL<br clear="none"><br clear="none"># have to find the right one, for me the first one in the list worked<br clear="none">CL-USER> (car (car (dump-classpath)))<br clear="none">#<org.armedbear.lisp.JavaClassLoader org.armedbear.lisp.JavaClassLoad.... {EF8EDD9}><br clear="none"><br clear="none">CL-USER> #1"Thread.currentThread()"<br clear="none">#<java.lang.Thread Thread[repl-thread,5,main] {16B905B3}><br clear="none"><br clear="none"># well, thread first, then class loader<br clear="none">CL-USER> (#"setContextClassLoader" * **)<br clear="none">NIL<br clear="none"><br clear="none"># looks like it works<br clear="none">CL-USER> #1"Thread.currentThread().getContextClassLoader().getResourceAsStream("spark-version-info.properties")"<br clear="none">#<sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream sun.net.www.protocol.jar.JarURLC.... {7B45ACFC}><br clear="none"><br clear="none">CL-USER> (defvar *spark-conf* #1"new SparkConf().setAppName("abcl-app").setMaster("local")" )<br clear="none">*SPARK-CONF*<br clear="none"><br clear="none"># important to only attempt this call last, otherwise it might throw<br clear="none">errors (c.f. *inferior-lisp*) about already having one in the process of<br clear="none">being constructed<br clear="none">CL-USER> (defvar *sc* (jss:new 'JavaSparkContext *spark-conf*))<br clear="none">*SC*<br clear="none">```<br clear="none"><br clear="none">Hopefully there's a better way of course, since this is hardly<br clear="none">convenient.<div class="yqt6600991273" id="yqtfd81668"><br clear="none"><br clear="none"></div></div>
</div>
</div></body></html>