Repository navigation
How to Manage and Organize CLI Tools? #559
Description
Since we are releasing a new version of biojava that introduces a number of API changes, I would like to suggest a review of some of the command line (CLI) tools that we have.
To my knowledge, we currently have CLI tools for the structural alignment algorithms and some symmetry analysis. The fist ones are implemented using a custom UserArgumentProcessor class. The second ones are implemented in a module of a separate repository called symmetry-tools.
The idea would be to have a more systematic way to create scripts (packaged as jar files) that can be distributed as standalone programs with the only requirement of java. An idea would be to split the algorithms (the computation code) and the command line interfaces (main methods) by creating a separate package called biojava-tools (for example), which depends on all the rest of biojava projects and special dependencies for the management of specific CLI tasks (like commons-cli).
Another possible option is to move the CLI scripts to a separate repository.
Activity
andreasprlic commentedon Aug 12, 2016
also the aa-prop module has a command line interface. see
org.biojava.nbio.aaproperties.CommandPrompt.andreasprlic commentedon Aug 12, 2016
also: protein-disorder:
org.biojava.nbio.ronn.ORonn.josemduarte commentedon Aug 12, 2016
I'd say a new module would be ideal for this.
I'd also support the use of commons-cli when moving to a separate module. In general it's a good idea to use standard libraries for this sort of things.
heuermh commentedon Aug 12, 2016
+1 to a new module.
Feel free to crib from this pom.xml, it uses templating-maven-plugin for access to build information at runtime and the appassembler plugin to create command line script wrappers. I prefer those to runnable jars.
I also prefer Args4j over commons-cli.
phidias51 commentedon Aug 12, 2016
I was wondering how the commands would be used?
Do users want to be able to have all commands from all modules accessible in a single tool, or on a module-by-module basis? You might also have a third option which looks at all commands from all of the modules that have been downloaded and then lists those commands.
Do users want to be able to pipe together commands? For example, you might have a command that downloads gene information from EntrezGene and another command that writes gene information into a biosql database or writes data into a JSON file.
Do users want a REPL-like experience, where you start a command-line app and then execute commands interactively? For example, you might have a command that runs PubMed queries and displays the results.
brianrepko commentedon Aug 12, 2016
for me jcommander > args4j > commons-cli - fwiw
jcommander and args4j are very similar but jcommander is more active in development
Java9 will have a built-in REPL by the way and one can also use bean shell (which I've done with BioJava way back when...)
heuermh commentedon Aug 12, 2016
Good questions, @phidias51!
I would propose the tool module depends on all the other modules, so all the cli classes and scripts would be in one place. There should also be a top level
biojava-toolsor such script that lists all the available sub-commands git-style. There is probably some way of doing that in bash but I did it in code here.All the cli classes should try to do reasonable things with stdin and stdout so they may be piped together.
For the REPL, as @brianrepko says, all we need is a cli class that puts everything in the classpath and then starts bsh or the Java9 thing if/when that is available.
phidias51 commentedon Aug 12, 2016
@heuermh Have you thought about using ServiceLoader to discover command implementations at runtime? This would let you have commands in different modules, and as long as the module was in the classpath then all commands from that module would be visible. This would make it possible for a user who's only interested in a single module to make use of it.
AFAIK jrunscript (in Java 1.8) can be used as a REPL.
In Groovy there's an annotation called @grab that lets you add a library to a script. When the script is executed, it downloads the library from maven central (or whatever repo you designate). I don't know if something like that will be in the forthcoming Java 9 release.
heuermh commentedon Aug 13, 2016
ServiceLoader seems like overkill to me, and if the command implementations were spread out across more than one module and discoverable only at runtime, then I don't see how the appassembler plugin stuff could work. That said, I wouldn't -1 if things went that way.
phidias51 commentedon Aug 13, 2016
@heuermh The appassembler would build the CLI skeleton application. When you start up the app, the skeleton app would check its classpath to discover all modules that have been downloaded, and use the ServiceLoader to discover all available implementations.
heuermh commentedon Aug 13, 2016
Ah, I see. Looks like this might be a case of whomever codes it first wins. 😉
sbliven commentedon Aug 15, 2016
I assume that the advantage to dynamically loading modules is to reduce the size of the download? In practice, it seems like most people are happy to accept larger file sizes if things work out of the box, rather than needing to muck around with classpath settings in scripts.
I've been favoring the shade approach, where you get a single executable jar file with all dependencies bundled in. Sure, its a bit larger, but it runs without any hassle. In theory, shade can even remove unused classes from the bundle, but I have yet to figure out how to prevent things like log4j from being pruned.
I haven't used appassembler before, but it looks like it sets up a wrapper script with the classpath set? If so, that might work better if executed in individual modules rather than biojava-tools, since you wouldn't want all dependencies included with all CLIs (if you're worried about size).
phidias51 commentedon Aug 15, 2016
These are all valid approaches. It would be useful to get some input from users.
1 remaining item
sbliven commentedon Aug 16, 2016
The nice thing about a shaded jar is that it's double-clickable (using Manifest Main-Class). Some of our tools make sense to use without CLI parameters, so this can be nice.
lafita commentedon Sep 12, 2016
There seems to be different opinions about how the biojava-tools can be organized and different implementation and library preferences. I guess we all agree that it would be beneficial to have a separation between the main algorithmic code and some easy to access tools for common types of applications. It would also be an advantage to have these tools ready to use, so that users do not have to get into the details of the Java code to use them, unless they want to change some very specific options not accessible from the CL interface (or participate in their development).
Perhaps we can define some basic rules that those tools must have, and then give freedom to anyone to choose the implementation and configuration details. Each tool then has to be independent from the others and must have a clear application (goal). By rules I am thinking about a proper documentation of the application of the tool, a clear description of the inputs, outputs and options of the tool and a clear example.
The biojava-tools will then be just an aggregator of those tools, with the only common denominator of using biojava code. We can create a separate repository named biojava-tools, where each tool can be in a different module (so that dependencies are chosen separately, they can even depend on a different version of biojava), and the final compiled jars can be stored in a single place, so that they are easy to find.
This is just an organizational suggestion, to allow more flexibility after all the possible options we have seen. Anyway, I will remove this issue from the BioJava 5 milestone, since it turned out to be more complicated than it seemed.
josemduarte commentedon Sep 12, 2016
I like the idea of a separate biojava-tools repository, that would be the most flexible solution. We could have as many modules as we wanted, with a tool per module.
phidias51 commentedon Sep 21, 2016
Thought this article on JShell, the new java REPL might be of interest...
Regards,
Mark
http://www.infoq.com/news/2016/09/JavaOne-2016-Keynote-JShell?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=global
On Sep 12, 2016 11:40 AM, "Jose Manuel Duarte" notifications@github.com
wrote:
sbliven commentedon Nov 25, 2016
Adding biojava to some major os repos, as @rmoszczynski suggests, would increase our visibility. I think biojava's primary use will be as a java library included via maven, but a monolithic executable that can run the basic algorithms would be quite useful.
josemduarte commentedon Mar 7, 2018
It would be important to extract the CLI tools out of the biojava code base, following the ideas discussed above. I'll mark this one as 5.1.0 milestone.
phidias51 commentedon Mar 10, 2018
@rmoszczynski The brew idea sounds interesting. There's an alternative though that's usable as an install tool from any OS that has bash, or Windows Powershell. It's called sdkman It's currently used to installed OpenJava, Groovy, Grails, and a wide variety of other programs. It has a neat feature that allows you to install multiple versions of a program, and switch between versions from the command line.