BigStitcher spark Dependency problem #26
Description
I was running BigStitcher Spark on a cluster single-node. At the end of first step "detect interesting point" its throwing an exceptions. Any suggestion ??
I have pasted logs below:
processing tpId=263 setupId=7, [724, 1536, 0] -> [1191, 1919, 46], dimensions (468, 384, 47) of full interval [724, 0, 0] -> [1191, 1919, 46], dimensions (468, 1920, 47)
Total number of jobs: 44571648
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
24/04/05 01:44:11 INFO SparkContext: Running Spark version 3.3.2
24/04/05 01:44:12 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
24/04/05 01:44:12 INFO ResourceUtils: ==============================================================
24/04/05 01:44:12 INFO ResourceUtils: No custom resources configured for spark.driver.
24/04/05 01:44:12 INFO ResourceUtils: ==============================================================
24/04/05 01:44:12 INFO SparkContext: Submitted application: SparkInterestPointDetection
24/04/05 01:44:12 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memunt: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
24/04/05 01:44:12 INFO ResourceProfile: Limiting resource is cpu
24/04/05 01:44:12 INFO ResourceProfileManager: Added ResourceProfile id: 0
24/04/05 01:44:13 INFO SecurityManager: Changing view acls to: salam
24/04/05 01:44:13 INFO SecurityManager: Changing modify acls to: salam
24/04/05 01:44:13 INFO SecurityManager: Changing view acls groups to:
24/04/05 01:44:13 INFO SecurityManager: Changing modify acls groups to:
24/04/05 01:44:13 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(salam); groups with view permissiondify permissions: Set()
24/04/05 01:44:13 INFO Utils: Successfully started service 'sparkDriver' on port 37185.
24/04/05 01:44:13 INFO SparkEnv: Registering MapOutputTracker
24/04/05 01:44:13 INFO SparkEnv: Registering BlockManagerMaster
24/04/05 01:44:13 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
24/04/05 01:44:13 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
Exception in thread "main" java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x69ecddc6) cannot access class sun.nio.ch.Direort sun.nio.ch to unnamed module @0x69ecddc6
at org.apache.spark.storage.StorageUtils$.(StorageUtils.scala:213)
at org.apache.spark.storage.StorageUtils$.(StorageUtils.scala)
at org.apache.spark.storage.BlockManagerMasterEndpoint.(BlockManagerMasterEndpoint.scala:114)
at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:353)
at org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:290)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:339)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:194)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:279)
at org.apache.spark.SparkContext.(SparkContext.scala:464)
at org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58)
at net.preibisch.bigstitcher.spark.SparkInterestPointDetection.call(SparkInterestPointDetection.java:276)
at net.preibisch.bigstitcher.spark.SparkInterestPointDetection.call(SparkInterestPointDetection.java:79)
at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
at picocli.CommandLine.access$1500(CommandLine.java:148)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
at picocli.CommandLine.execute(CommandLine.java:2170)
at net.preibisch.bigstitcher.spark.SparkInterestPointDetection.main(SparkInterestPointDetection.java:758)**