6. Optional Command Line Interface Usage
7. Change web application properties or name
9. Information for developers
10. Tuning OpenGrok for large code bases
OpenGrok is a fast and usable source code search and cross reference
engine, written in Java. It helps you search, cross-reference and navigate
your source tree. It can understand various program file formats and
version control histories like SCCS, RCS, CVS, Subversion, Mercurial etc.
Offical page of the project is on:
* Latest Java (At least 1.7, 1.8 is supported)
* A servlet container like Tomcat (
7.x or later)
supporting Servlet 2.4 and JSP 2.0
* Source Code Management installation
depending on type of repositories indexed
* If you want to build OpenGrok:
- Netbeans (optional, at least 8.0, will need Ant 1.9.3)
OpenGrok usually runs in servlet container (
e.g. Tomcat).
SRC_ROOT environment variable refers to the directory containing your source
tree. OpenGrok analyzes the source tree and builds a search index along with
cross-referenced hypertext versions of the source files. These generated
data files will be stored in directory referred to with environment variable
OpenGrok has a concept of Projects - one project is one directory underneath
SRC_ROOT directory which usually contains a checkout of a project sources.
(this can be branch, version, ...)
Projects effectively replace the need to have more web applications, each with
opengrok .war file. Instead it leaves you with one indexer and one web
application serving multiple source code repositories - projects.
Then you have a simple update script and simple index refresher script in
place, which simplifies management of more repositories.
A nice concept is to have a naming convention for directories underneath
SRC_ROOT, thereby creating a good overview of projects (
e.g.For example, the SRC_ROOT directory can contain the following directories:
Each of these directories was created with 'cvs checkout' command (with
appropriate arguments to get given branch) and will be treated by OpenGrok
4.1 Installing on Solaris from *.p5p file
The file <package_name>.p5p you can easily use as a new publisher for the pkg command.
# pkg install --no-refresh -g
/path/to/file/<package_name>.p5p opengrok
You can also update OpenGrok software with the *.p5p file by running a command
# pkg update --no-refresh -g
/path/to/file/<package_name>.p5p 'pkg://<publisher>/*'
To setup OpenGrok it is needed to prepare the source code, let OpenGrok index
it and start the web application.
5.1 Setting up the sources
--------------------------
Source base should be available locally for OpenGrok to work efficiently.
No changes are required to your source tree. If the code is under source
control management (SCM) OpenGrok requires the checked out source tree under
By itself OpenGrok does not perform the setup of the source code repositories
or sychronization of the source code with its origin. This needs to be done by
the user or by using automatic scripts.
It is possible for SCM systems which are not distributed (Subversion, CVS)
to use a remote repository but this is not recommended due to the performance
penalty. Special option when running the OpenGrok indexer is needed to enable
remote repository support ("-r on").
In order for history indexing to work for any SCM system it is necessary
to have environment for given SCM systems installed and in a path accessible
Note that OpenGrok ignores symbolic links.
If you want to skip indexing the history of a particular directory
(and all of it's subdirectories), you can touch '.opengrok_skip_history' file
at the root of that directory.
5.2 Using Opengrok wrapper script to create indexes
---------------------------------------------------
For *nix systems there is a shell script called OpenGrok which simplifies most
of the tasks. It has been tested on Solaris and Linux distributions.
5.2.1 - Deploy the web application
----------------------------------
First please change to opengrok directory where the OpenGrok shell script is
stored (can vary on your system).
Note that now you might need to change to user which owns the target
directories for data,
e.g. on Solaris you'd do:
This command will do some sanity checks and will deploy the
source.war in
its directory to one of detected web application containers.
Please follow the error message it provides.
If it fails to discover your container, please refer to optional steps on
changing web application properties below, which explains how to do this.
Note that OpenGrok script expects the directory
/var/opengrok to be
available to user running opengrok with all permissions. In root user case
it will create all the directories needed, otherwise you have to manually
create the directory and grant all permissions to the user used.
5.2.2 - Populate DATA_ROOT Directory
------------------------------------
During this process the indexer will generate OpenGrok XML configuration file
The indexing can take a lot of time. After this is done, indexer automatically
attempts to upload newly generated configuration to the web application.
Most probably you will not be able to use Opengrok before this is done for the
Please change to opengrok directory (can vary on your system)
otherwise (if SRC_ROOT is in different directory) run:
$ ./OpenGrok index <absolute_path_to_your_SRC_ROOT>
The above command attempts to upload the latest index status reflected into
Once above command finishes without errors
(
e.g. SEVERE: Failed to send configuration to localhost:2424),
you should be able to enjoy your opengrok and search your sources using
latest indexes and setup.
It is assumed that any SCM commands are reachable in one of the components
of the PATH environment variable (
e.g. 'git' command for Git repositories).
Likewise, this should be maintained in the environment of the user which runs
Congratulations, you should now be able to point your browser to
http://<YOUR_WEBAPP_SERVER>:<WEBAPPSRV_PORT>/source to work with your fresh
OpenGrok installation! :-)
At this time we'd like to point out some customization to OpenGrok script
A common case would be, that you want the data in some other directory than
/var/opengrok. This can be easily achieved by using environment variable
in
/tank/source then to get more verbosity run the indexer as:
Since above will also change default location of config file, beforehands(or
restart your web container after creating this symlink) I suggest doing
More customizations can be found inside the script, you just need to
have a look at it, eventually create a configuration out of it and use
OPENGROK_CONFIGURATION environment variable to point to it. Obviously such
setups can be used for nightly cron job updates of index or other automated
5.3 Using SMF service (Solaris) to maintain OpenGrok indexes
------------------------------------------------------------
If you installed OpenGrok from the OSOLopengrok package, it will work out of
the box. Should you need to configure it (
e.g. because of non-default SRC_ROOT
or DATA_ROOT paths) it is done via the 'opengrok' property group of the
# svccfg -s opengrok setprop \
Then make the service start the indexing, at this point it would be nice if
the web application is already running.
# svcadm enable -rs opengrok
Note that this will enable tomcat6 service as dependency.
When the service starts indexing for first time, it's already enabled and
depending on tomcat6, so at this point the web application should be
Note that indexing is not done when the opengrok service is disabled.
To rebuild the index later (
e.g. after source code changed) just run:
# svcadm refresh opengrok
The service makes it possible to supply part of the configuration via the
Note: before removing the package please disable the service.
If you don't do it, it will not be removed automatically.
In such case please remove it manually.
5.4 Using command line interface to create indexes
--------------------------------------------------
There are 2 (or 3) steps needed for this task.
5.4.1 - Populate DATA_ROOT Directory
------------------------------------
Option 1. OpenGrok: There is a sample shell script OpenGrok that is suitable
for using in a cron job to run regularly. Modify the variables in the script
to point appropriate directories, or as the code suggests factor your local
configuration into a separate file and simplify future upgrades.
Option 2.
opengrok.jar: You can also directly use the Java application. If
the sources are all located in a directory SRC_ROOT and the data and
hypertext files generated by OpenGrok are to be stored in DATA_ROOT, run
----------------------------------------------
To configure the webapp
source.war, look into the parameters defined in
* HEADER: is the fragment of HTML that will be used to display title or
* SRC_ROOT: absolute path name of the root directory of your source tree
* DATA_ROOT: absolute path of the directory where OpenGrok data
- Header file 'header_include' can be created under DATA_ROOT.
The contents of this file file will be appended to the header of each
web page after the OpenGrok logo element.
- Footer file 'footer_include' can be created under DATA_ROOT.
The contents of this file file will be appended to the footer of each
web page after the information about last index update.
5.4.3 - Path Descriptions (optional)
------------------------------------
OpenGrok can use path descriptions in various places (
e.g. while showing
directory listings or search results). Example descriptions are in
paths.tsvThe
paths.tsv file is read by OpenGrok indexing script from the configuration
dtags.eftar in the index subdirectory under DATA_ROOT directory which will
then be used by the webapp to display the descriptions.
The file contains descriptions for directories one per line. Path to the
directory and its description are separated by tab. The path to the directory
is absolute path under the SRC_ROOT directory.
For example, if the SRC_ROOT directory contains the following directories:
then the
paths.tsv file contents can look like this:
Note that only some paths can have a description.
5.4.4 - Changing webapp parameters (optional)
---------------------------------------------
web.xml is the deployment descriptor for the web application. It is in a Jar
file named
source.war, you can change it as follows:
edit
web.xml and re-package the jar file.
Then copy the war files to <i>TOMCAT</i>/webapps directory.
* Option 3: Edit the Context container element for the webapp
When invoking OpenGrok to build the index, use -w <webapp> to set the
context. If you change this(or set using OPENGROK_WEBAPP_CONTEXT) later,
FULL clean reindex is needed.
After the index is built, there's a couple different ways to set the
Context for the servlet container:
<Parameter name="HEADER" value='...' override="false" />
- Create a Context file for the webapp
This file will be named `<webapp>.xml'.
For Tomcat, the file will be located at:
`
TOMCAT/conf/<engine_name>/<hostname>', where <engine_name>
is the Engine that is processing requests and <hostname> is a Host
associated with that Engine. By default, this path is
This file will contain something like the Context described above.
5.4.5 Custom ctags configuration
--------------------------------
specify configuration file with extra configuration options for ctags.
This can be done by setting OPENGROK_CTAGS_OPTIONS_FILE environment variable
when running the OpenGrok shell script (or directly with the -o option for
opengrok.jar). Default location for the configuration file in the OpenGrok
Sample configuration file for Solaris code base is delivered in the doc/
5.5 Using Java DB for history cache
-----------------------------------
By default OpenGrok stores history indexes in gzipped xml files. An alternative
is to use Java DB instead. This has the advantage of less disk space and
incremental indexing. Also, for some Source Code Management systems the
History view will show a list of files modified with given change.
On the other hand it consumes more system memory because the database has to
Versions of Java DB from 10.5.3 to 10.8.3.0 are known to work.
Java DB
10.10.x.y versions are known to NOT work.
To install it perform the following steps:
# apt-get install sun-java6-javadb
Fetch the db-derby bundle from
db.apache.org, and unpack to your preferred path.
There are two modes, having Java DB embedded, or running a Java DB server.
Java DB server is the default option, we will not describe how to set up the
Use one of the following methods to start the database:
E.g. for Tomcat it is located in the WEB-INF directory which was created
For example on Solaris 11 with bundled Java DB and Tomcat and OpenGrok
installed from the OSOLopengrok package the command will be:
This is achieved by setting the OPENGROK_DERBY environment variable when
using the OpenGrok shell script.
The Java DB server has to be running during indexing and for the web
Note: To use a bigger database buffer, which may improve performance of both
the JavaDB data directory and add this line to it:
- If using specific data directory:
- Using default directory on Solaris with JavaDB being run from SMF:
5.6 Introduce own mapping for an extension to analyzer
------------------------------------------------------
OpenGrok script doesn't support this out of box, so you'd need to add it there.
Usually to StdInvocation() function after line -jar ${OPENGROK_JAR} .
(this will map extension .cs to PlainAnalyzer)
You should even be able to override OpenGroks analyzers using this option.
6. Optional Command Line Interface Usage
----------------------------------------
You need to pass location of project file + the query to Search class,
e.g.7. Change web application properties or name
--------------------------------------------
You might need to modify the web application if you don't store the
configuration file in the default location
To configure the webapp
source.war, look into the parameters defined in
parameters appropriately. These sample parameters need modifying(there are
more options, refer to manual or read param comments).
* CONFIGURATION - the absolute path to XML file containing project
* ConfigAddress - port for remote updates to configuration, optional,
* but advised(since there is no authentication) to be set to
* localhost:<some_port> (
e.g. localhost:2424), if you choose some_port
* below 1024 you have to have root privileges
If you need to change name of the web application from source to something
else you need to use special option -w <new_name> for indexer to create
proper xrefs, besides changing the .war file name. Be sure that when this
changed you reindex cleanly from scratch. Examples below show just
* Option 1: Use browser and log into glassfish web administration interface
Common Tasks / Applications / Web Applications , button Deploy and point
to deploy it "auto magically".
* Option 3: Use cli from GLASSFISH directory:
Deploy the modified .war file in tomcat:
The indexer can be setup with agent and systray GUI control application.
This is optional step for those who wish to monitor and configure OpenGrok
from their desktop using systray application.
starting special OpenGrok Agent, where you can connect with a systray GUI
To start the indexer with configuration run:
Then from the remote machine one can run:
assuming configuration permits remote connections (
i.e. not listening on
localhost, but rather on a physical network interface).
This agent is work in progress, so it might not fully work.
9. Information for developers
-----------------------------
Just run 'ant' from command line in the top-level directory or use build
process driven by graphical developer environment such as Netbeans.
Note: in case you are behind http proxy, use ANT_OPTS to download jflex, lucene
Run 'ant package' to create package (specific for the operating system this is
being executed on) under the dist/ directory.
Note: For full coverage report your system has to provide proper junit test
environment, that would mean:
- you have to use Ant 1.7 and above
- Example install in the top of the opengrok repository:
- install
derby.jar to ant's classpath so that Java DB tests can be run
- your PATH must contain directory with exuberant ctags binary
- Note: make sure that the directory which contains exuberant ctags binary
is prepended before the directory with plain ctags program.
- your PATH variable must contain directories which contain binaries of
appropriate SCM software which means commands hg, sccs, cvs, git, bzr, svn
(svnadmin too). They must be available for the full report.
The tests are then run as follows:
To check if the test completed without error look for AssertionFailedError
you have to download Findbugs to your machine, and install it where you have
checked out your OpenGrok source code, under the
lib/findbugs directory,
You can now run ant with the findbugs target:
[findbugs] Executing findbugs from ant task
[findbugs] Running FindBugs...
[findbugs] Warnings generated: nnn
If you want to install findbugs some other place than ~/.ant/lib, you can
tell ant where to find findbugs, like this (if you have installed fundbugs
under the lib directory):
There is also a findbugs-xml ant target that can be used to generate XML files
that can later be parsed,
e.g. by Jenkins.
If you want to check test coverage on OpenGrok, download Emma from
Now you can instrument your classes, and create a jar file:
If you are using NetBeans, select File - "opengrok" Properties
- libraries - Compile tab. Press the "Add
JAR/Folder" and select
If you are not using netbeans, you have to edit the file
Now you can put the classes into jars and generate distributable:
the run time analysis, use these properties:
directory for easy analyze.
If you want to test the coverage of the unit tests, you can
Alternatively press Alt+F6 in NetBeans to achieve the same.
Now you should get some output saying that Emma is placing runtime
To generate reports, run ant again:
To check that your code follows the standard coding conventions,
You need Version 5.3 (or newer). Extract the package you have
$ ln -s checkstyle-5.3 checkstyle
You also have to create symbolic links to the jar files:
To run checkstyle on the source code, just run ant checkstyle:
Output from the command will be stored in the checkstyle directory.
If you want to install checkstyle some other place than ~/.ant/lib, you can
to tell ant where to find checkstyle, like this (if you have installed
checkstyle under the lib directory):
To check the quality of the OpenGrok code you can also use PMD
You also have to make links to the jar files:
To run PMD on the rource code, just run ant pmd:
Outout from the command will be stored in the pmd subdirectory:
If you want to install PMD some other place than ~/.ant/lib, you can
unzip the .zip file to a directory, and use the
pmd.home property
to tell ant where to find PMD, like this (if you have installed
PMD under the lib directory):
To run CPD, just use the same as above, but use targets:
To see dependencies in the source code, you can use JDepend from
$ ln -s jdepend-2.9/ jdepend
Output is stored in the jdepend directory:
$ export SONAR_RUNNER_OPTS="-Xmx768m -XX:MaxPermSize=256m"
$ export SERVERIP=10.163.26.78
you should be able to connect your Github to Travis CI.
10. Tuning OpenGrok for large code bases
---------------------------------------
While indexing big source repos you might consider using ZFS filesystem to give
you advantage of datasets which can be flipped over or cloned when needed.
If the machine is strong enough it will also give you an option to
incrementally index in parallel to having the current sources and index in sync.
(So tomcat sees certain zfs datasets, then you just stop it, flip datasets to
the ones that were updated by
SCM/index and start tomcat again - outage is
minimal, sources+indexes are ALWAYS in sync, users see the truth)
OpenGrok script by default uses 2G of heap and 16MB per thread for flush size of
lucene docs indexing(when to flush to disk).
It also uses default 32bit JRE.
This MIGHT NOT be enough. You might need to consider this:
Lucene
4.x sets indexer defaults:
DEFAULT_RAM_PER_THREAD_HARD_LIMIT_MB = 1945;
DEFAULT_MAX_THREAD_STATES = 8;
DEFAULT_RAM_BUFFER_SIZE_MB = 16.0;
- which might grow as big as 16GB (though DEFAULT_RAM_BUFFER_SIZE_MB shouldn't
really allow it, but keep it around 1-2GB)
- the lucenes RAM_BUFFER_SIZE_MB can be tuned now using the parameter -m, so
running a 8GB 64 bit server JDK indexer with tuned docs flushing(on Solaris 11):
# export JAVA_OPTS="-Xmx8192m -server"
# OPENGROK_FLUSH_RAM_BUFFER_SIZE="-m 256" ./OpenGrok index /source
Tomcat by default also supports only small deployments. For bigger ones you
MIGHT need to increase its heap which might necessitate the switch to 64-bit
Java. It will most probably be the same for other containers as well.
JAVA_OPTS="$JAVA_OPTS -d64 -server"
# OpenGrok memory boost to cover all-project searches
# (7 MB * 247 projects + 300 MB for cache should be enough)
# 64-bit Java allows for more so let's use 8GB to be on the safe side.
# We might need to allow more for concurrent all-project searches.
JAVA_OPTS="$JAVA_OPTS -Xmx8g"
For tomcat you might also hit a limit for http header size (we use it to send
the project list when requesting search results):
connectionTimeout="20000"
maxHttpHeaderSize="65536"
Refer to docs of other containers for more info on how to achieve the same.
The project has been originally conceived in Sun Microsystems by Chandan
B.N.Martin Englund, (originally Sun Microsystems)
Feel free to participate in discussion on discuss@opengrok.java.net.
You can subscribe via web interface on: