JavaPackage'object is not callable

[radi2015]

in Getting Started
python3 -m pip install pyrasterframes
import pyrasterframes
from pyrasterframes.utils import create_rf_spark_session
spark = create_rf_spark_session()
Error setting up SparkSession; cannot find the pyrasterframes assembly jar
'JavaPackage' object is not callable

I haven't installed the pirasterframes for a day

[pomadchin]

Hey @radi2015, I checked the same steps in a fresh conda environment:

name: rf
channels:
 - conda-forge
dependencies:
 - python=3.9.10
 - pyrasterframes

And the output that I see:

$ python3         
Python 3.9.10 | packaged by conda-forge | (main, Feb  1 2022, 21:28:27) 
[Clang 11.1.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyrasterframes
>>> from pyrasterframes.utils import create_rf_spark_session
>>> spark = create_rf_spark_session()
2022-04-02 00:07:56,589 WARN util.Utils: Your hostname, macbook.local resolves to a loopback address: 127.0.0.1; using 10.0.0.224 instead (on interface en0)
2022-04-02 00:07:56,590 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind to another address
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/Users/.../miniconda/envs/geoparquet/lib/python3.9/site-packages/pyspark/jars/spark-unsafe_2.12-3.1.2.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2022-04-02 00:07:57,056 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
>>> 

I will make an assumption that or you don't have a properly installed Java or you have some other specifics of the infra you're using (i.e. Windows) and not sharing in the issue description, please try to follow the Getting Started guide one more time; to write a more comprehensive response I need to know more, since as I said, the setup in a fresh conda env just works for me; in the worst case you may try it in a prepared Docker env.

We're open to PRs, and if you'd like to improve docs, you're more than welcome.

I'd also like to remind you that is an opensource project, and people on this project often volunteering their time to maintain it. If you have any questions / issues feel free to reach us here or in the gitter channel for the interactive chat; but all questions / comments related to other projects please redirect to Stackoverflow / other projects repositories instead.

[radi2015]

Sorry, there is something wrong with my words. This is a great project. I really want to learn to use it. Maybe there is something wrong with my operation.

In the Python interpreter above you, I saw how to start, and I succeeded
I'll tell you what I'm wrong

pyspark
Python 3.7.11 (default, Jul 27 2021, 07:03:16) 
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
22/04/02 13:58:26 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
import pyrasterframesWelcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 3.1.3
      /_/

Using Python version 3.7.11 (default, Jul 27 2021 07:03:16)
Spark context Web UI available at http://10.30.108.108:4040
Spark context available as 'sc' (master = local[*], app id = local-1648879113934).
SparkSession available as 'spark'.
>>> import pyrasterframes
>>> from pyrasterframes.utils import create_rf_spark_session
>>> spark = create_rf_spark_session()
Error setting up SparkSession; cannot find the pyrasterframes assembly jar
 'JavaPackage' object is not callable
>>> 

I shouldn't start pyspark first, I should start Python first

thank you very much

[pomadchin]

@radi2015 great that it works now! how have you tried it before and what is the difference?

In this case Spark session creation makes your Python interpreter process a driver;

if you want to try the same in the spark-shell, I think you’d need to configure the class path a bit, so it knows where to find Python libs as well as Java jars.

[radi2015]

At the beginning, I started the pyspark command first, and then the sparksession was created. Then I executed the create_rf_spark_session() function and reported an error. When I saw the error, I thought it was the jar package of rasterframe that was not put into the Lib directory of spark. If I put the jar package into the Lib directory, it also report an error. Now, it seems that the reason is that if pyspark cmd create a sparksession first,then executing the create_rf_spark_session() function create spark session again will be error。

[pomadchin]

@radi2015 oh it makes sense, I think you'd need to try smth like spark.withRasterFrames() while being in the spark-shell to make it work; indeed it is impossible to create several sessions within the same jvm

[mathiaskvist]

I have the same issue when trying to install on Databricks. When installing pyrasterframes==0.10.1 on the cluster it is successful, but when I go to the notebook I get the error:

>>> import pyrasterframes
>>> spark = spark.withRasterFrames()
TypeError: 'JavaPackage' object is not callable

Also, it is unclear to me based on the documentation, whether it is necessary to install these java packages in order to use pyrasterframes:

org.locationtech.rasterframes:rasterframes_2.12:0.10.1
org.locationtech.rasterframes:pyrasterframes_2.12:0.10.1
org.locationtech.rasterframes:rasterframes-datasource_2.12:0.10.1

The installation of the last two packages fail with the error:

Library resolution failed. Cause: java.lang.RuntimeException: unresolved dependency: com.github.everit-org.json-schema:org.everit.json.schema:1.12.2: not found

[pomadchin]

hey @mathiaskvist that's due to stac4s in deps and due to the json schema validation library we use, add the following https://jitpack.io URI as an extra resolver.