Include Native Libaries in ScalaTask

Streetworker · 2017-11-02 13:54:25 UTC

To run the code in a common JVM, I just call java -jar simulation.jar.
This way I don’t have to add the lib-location to the os native search path.
Adding the library files inside the jar is not possible.
My first approach was to add the files as parameter “inputs +=” in openmole. But here the ScalaTask can’t find the library files.
The second approach was to make use of the parameter “libraries +=”, but this also fails (“error while loading , Error accessing…”).
Is there any chance to copy the lib-files to the same directory as the jar file?
Or do I have to change my program and put it into a native code package, such as CARETask or SystemExecTask ?

Thanks
Streetworker

reuillon · 2017-11-02 19:47:30 UTC

Could you post the java code that is used to load the shared lib?

Streetworker · 2017-11-03 00:52:17 UTC

The shared library is called spatialite. It’s an extension for sqlite and adds spatial features.
In Java this exension is typcially loaded dynamically via a JDBC driver. Here an example from https://www.gaia-gis.it/gaia-sins/spatialite-cookbook/html/java.html

        // enabling dynamic extension loading
        // absolutely required by SpatiaLite
        SQLiteConfig config = new SQLiteConfig();
        config.enableLoadExtension(true);

        // create a database connection
        conn = DriverManager.getConnection("jdbc:sqlite:spatialite-test.sqlite",
        config.toProperties());
        Statement stmt = conn.createStatement();

        // loading SpatiaLite
        stmt.execute("SELECT load_extension('mod_spatialite')");

        // checking SQLite and SpatiaLite version + target CPU
        String sql = "SELECT sqlite_version(), spatialite_version(), spatialite_target_cpu()";
        ResultSet rs = stmt.executeQuery(sql);
        while(rs.next()) {
            // read the result set
            String msg = "SQLite version: ";
            msg += rs.getString(1);
            System.out.println(msg);
        }

The SELECT load_extension requires the name of the shared library (.dll/.so/.dylib-file).
During runtime, the extension loader searches for the given filename in the os search path and the working directory. Since in my case, I don’t want to use the path environment, I have to choose the later option.
If you need more information, please just ask.

reuillon · 2017-11-03 10:36:23 UTC

The working dir is setup for the whole process (in this case the JVM). We can not change it for each Scala task execution. Do you have any way to pass the directory as an argument (by providing an absolute path to load extension for instance or by using a method of execute)?

After searching it seems that a absolute path works: http://www.gaia-gis.it/spatialite-2.4.0-4/splite-jdbc.html

In this case you can do something like this:


val scalaTask = ScalaTask("""val res = callMyFunction(arg1, arg2,  workDirectory / "mod_spatialite.so")""") set (
  resources += workDirectory / "mod_spatialite.so"
)

The signature of callMyFunction should be:

def callMyFunction(arg1: arg1Type, arg2: arg2Type, lib: File): retType

The resources will be deployed at runtime in a directory which is accessible in the task using the workDirectory variable. This should work but is completely undocumented in the current doc.

Streetworker · 2017-11-04 10:19:42 UTC

Unfortunately, this issue isn’t solvable that easy. Your idea doesn’t work at a cluster use-case since the filename extension created by openmole is .bin and not the original filename.
However the sqlite extension loader requires os depended extensions such as .so/.dll/.dylib. If the filename does’t show this extension, it automatically adds it. This behavior results in following error message:

Lib File is: filee245afa5-7a0d-4164-949d-7f1f7a9bb2e1.bin
LoadLibrary function called: /home/user/.openmole/.tmp/ssh/openmole-45b65910-f3ce-4fcd-93b6-b4ef26ff4705/tmp/1509788696199/247d8a18-1228-40e3-95d7-1f964a333beb/e6bf9975-9ed5-4b66-9291-4ee2c3ad88c7/.tmp/2166fe3b-1a6b-48e2-9d4b-1956266442ed/filee245afa5-7a0d-4164-949d-7f1f7a9bb2e1.bin

---------------------------Error output node12.cluster--------------------------
[SQLITE_ERROR] SQL error or missing database (filee245afa5-7a0d-4164-949d-7f1f7a9bb2e1.bin.so: cannot open shared object file: No such file or directory)

Do you have any idea how to solve this issue? For the local environment in windows, it works. If I execute the oms script several time in the same session, I get a following error message:

Caused by: java.lang.UnsatisfiedLinkError: Native Library C:\Users\user.openmole\PC\webui\projects\projectTest\omtestsimple\libproj-9.dll already loaded in another classloader
As the error says the shared libraries are already loaded by the JVM.

In all cases, it is needed to call the System.load(filename.getAbsolutePath()) function to load all depended shared libraries in memory.
Maybe it would help to add the working directory temporary to PATH/LD_LIBRARY_PATH as described in https://bitbucket.org/xerial/sqlite-jdbc/issues/187/cant-load-spatial-extension. Is this possible with openmole?

Here my script/java code:

Linux Cluster:

val i = Val[Int]
val o = Val[Int]
val libfreexl = Val[File]
val libgeos_c = Val[File]
val libgeos = Val[File]
val libproj = Val[File]
val mod_spatialite = Val[File]

val explo = ExplorationTask(i in (0 to 0))

val javaTask = ScalaTask(""“
val o = my.pkg.test.Main.run(i,libfreexl, libgeos_c, libgeos, libproj, mod_spatialite)
”"") set (
libraries += workDirectory / “omtest.jar”,
libraries += ((workDirectory / “libs”).listFiles.toSeq: _*),
resources += workDirectory / “libfreexl.so.1”,
resources += workDirectory / “libgeos_c.so.1”,
resources += workDirectory / “libgeos-3.5.1.so”,
resources += workDirectory / “libproj.so.12”,
resources += workDirectory / “mod_spatialite.so”,
inputs += (i,libfreexl,libgeos_c,libgeos,libproj,mod_spatialite),
outputs += o,
libfreexl := workDirectory / “libfreexl.so.1”,
libgeos_c := workDirectory / “libgeos_c.so.1”,
libgeos := workDirectory / “libgeos-3.5.1.so”,
libproj := workDirectory / “libproj.so.12”,
mod_spatialite := workDirectory / “mod_spatialite.so”
)

val env = PBSEnvironment(
“user”,
“1.1.1.1”)

val displayHook = ToStringHook(o)

explo -< (javaTask on env hook displayHook)

Windows Local:

val i = Val[Int]
val o = Val[Int]
val libfreexl = Val[File]
val libgcc_s_seh_64 = Val[File]
val libgcc_s_seh = Val[File]
val libgeos_c = Val[File]
val libgeos = Val[File]
val libiconv = Val[File]
val liblzma = Val[File]
val libproj = Val[File]
val libsqlite3 = Val[File]
val libstdc = Val[File]
val libxml2 = Val[File]
val mod_spatialite = Val[File]
val zlib1 = Val[File]

val explo = ExplorationTask(i in (0 to 0))

//Defines the task to perform the hello function
val javaTask = ScalaTask(""“
val o = my.pkg.test.Main.run(i,libfreexl, libgcc_s_seh_64, libgcc_s_seh, libgeos_c, libgeos, libiconv, liblzma, libproj, libsqlite3, libstdc, libxml2, mod_spatialite, zlib1)
”"") set (
libraries += workDirectory / “omtest.jar”,
libraries += ((workDirectory / “libs”).listFiles.toSeq: _*),
resources += workDirectory / “libfreexl-1.dll”,
resources += workDirectory / “libgcc_s_seh_64-1.dll”,
resources += workDirectory / “libgcc_s_seh-1.dll”,
resources += workDirectory / “libgeos_c-1.dll”,
resources += workDirectory / “libgeos-3-5-0.dll”,
resources += workDirectory / “libiconv-2.dll”,
resources += workDirectory / “liblzma-5.dll”,
resources += workDirectory / “libproj-9.dll”,
resources += workDirectory / “libsqlite3-0.dll”,
resources += workDirectory / “libstdc++_64-6.dll”,
resources += workDirectory / “libxml2-2.dll”,
resources += workDirectory / “mod_spatialite.dll”,
resources += workDirectory / “zlib1.dll”,
inputs += (i,libfreexl,libgcc_s_seh_64,libgcc_s_seh,libgeos_c,libgeos,libiconv,liblzma,libproj,libsqlite3,libstdc,libxml2,mod_spatialite,zlib1),
outputs += o,
libfreexl := workDirectory / “libfreexl-1.dll”,
libgcc_s_seh_64 := workDirectory / “libgcc_s_seh_64-1.dll”,
libgcc_s_seh := workDirectory / “libgcc_s_seh-1.dll”,
libgeos_c := workDirectory / “libgeos_c-1.dll”,
libgeos := workDirectory / “libgeos-3-5-0.dll”,
libiconv := workDirectory / “libiconv-2.dll”,
liblzma := workDirectory / “liblzma-5.dll”,
libproj := workDirectory / “libproj-9.dll”,
libsqlite3 := workDirectory / “libsqlite3-0.dll”,
libstdc := workDirectory / “libstdc++_64-6.dll”,
libxml2 := workDirectory / “libxml2-2.dll”,
mod_spatialite := workDirectory / “mod_spatialite.dll”,
zlib1 := workDirectory / “zlib1.dll”
)

val displayHook = ToStringHook(o)

val env = LocalEnvironment(1)

explo -< (javaTask on env hook displayHook)

JAVA Code

//Windows Config
public static int run(int i, File libfreexl, File libgcc_s_seh_64, File libgcc_s_seh, File libgeos_c, File libgeos, File libiconv, File liblzma, File libproj, File libsqlite3, File libstdc, File libxml2, File mod_spatialite, File zlib1) {
try {
retEnv();
loadLibWindows(libfreexl, libgcc_s_seh_64, libgcc_s_seh, libgeos_c, libgeos, libiconv, liblzma, libproj, libsqlite3, libstdc, libxml2, mod_spatialite, zlib1);

        String libName = mod_spatialite.getName();
        System.out.println(libName);
        SpatialiteSample.run(libName); // should ("mod_spatialite.dll");

    } catch (ClassNotFoundException e) {
        e.printStackTrace();
    }
    return 0;
}

private static void loadLibWindows(File libfreexl, File libgcc_s_seh_64, File libgcc_s_seh, File libgeos_c, File libgeos, File libiconv, File liblzma, File libproj, File libsqlite3, File libstdc, File libxml2, File mod_spatialite, File zlib1) {

    System.load(libproj.getAbsolutePath());
    System.load(libiconv.getAbsolutePath());
    System.load(libfreexl.getAbsolutePath());
    System.load(libgcc_s_seh.getAbsolutePath());
    System.load(libgcc_s_seh_64.getAbsolutePath());
    System.load(libstdc.getAbsolutePath());
    System.load(libgeos.getAbsolutePath());
    System.load(libgeos_c.getAbsolutePath());
    System.load(libsqlite3.getAbsolutePath());
    System.load(liblzma.getAbsolutePath());
    System.load(zlib1.getAbsolutePath());
    System.load(libxml2.getAbsolutePath());
    System.load(mod_spatialite.getAbsolutePath());

    return;
}

//Linux Config
public static int run(int i, File libfreexl, File libgeos_c, File libgeos, File libproj, File mod_spatialite) {
    try {
        retEnv();
        String libName = mod_spatialite.getName();
        System.out.println("Lib Name is " + libName);
        loadLibLinux(libfreexl, libgeos_c, libgeos, libproj, mod_spatialite);
        SpatialiteSample.run(libName);
    } catch (ClassNotFoundException e) {
        e.printStackTrace();
    }
    return 0;
}

private static void loadLibLinux(File libfreexl, File libgeos_c, File libgeos, File libproj, File mod_spatialite) {

    System.load(libfreexl.getAbsolutePath());
    System.load(libgeos.getAbsolutePath());
    System.load(libgeos_c.getAbsolutePath());
    System.load(libproj.getAbsolutePath());
    System.load(mod_spatialite.getAbsolutePath());
    //System.out.println("Lib loaded " + mod_spatialite.getAbsolutePath());

    return;
}

reuillon · 2017-11-04 12:13:03 UTC

For the naming part, this is strange since the resource mechanism is supposed to preserve the name. I think you shouldn’t use the file variable with default value (libfreexl := workDirectory / "libfreexl-1.dll" for instance), that’s where you get the .bin from. If you use the resource += file along with workDirectory / name in the Scala task you should get the correct name for the lib.

For the global JVM variable, you could just ignore the exception if it is already loaded.

That a bit dirty but it should work. I keep this use case in mind to see if I can come up with a cleaner solution… An idea I can think of is to let the user personalise some option for the runtime and link it to an environment. For instance you could set some prerun command like loading shared libs or setting env variables in the remote runtime before OpenMOLE starts running the tasks.

Streetworker · 2017-11-04 12:27:09 UTC

The reason why I introduce all these additional input parameters is that the ScalaTask execution otherwise fails:

org.openmole.core.exception.UserBadDataError: Formal validation of your mole has failed, 1 error(s) has(ve) been found.
Errors in validation of task javaTask@902305906:
| org.openmole.core.console.ScalaREPL$CompilationError: (line 14) not found: value workDirectory
| val o = my.pkg.test.Main.run(i,workDirectory / “libfreexl-1.dll”,…

It seems that the workDirectory variable can’t be resolved in a Scala Task. The signature looks like this:
val o = my.pkg.test.Main.run(i,workDirectory / “libfreexl-1.dll”,…) set (…)

reuillon · 2017-11-04 13:14:14 UTC

It is a bug. I just fixed it. Jenkins is building the fixed version right now. It should be available in 20 minutes here:
https://jenkins.iscpif.fr/job/openmole-master/lastSuccessfulBuild/artifact/openmole/bin/target/site/openmole.tar.gz