Rule #1: do not set JAVA_HOME unless you are an expert. Generate one (which involves compilation of Java code). Since the checkout is not the actual R package but a source to Package from the source checkout, you must use sh mkdist to do so Since rJava includes REngine as a submodule.