Omega overview
==============

Omega operates on a set of databases.  Each database is created and updated
separately using either omindex or scriptindex.  You can search these databases
(or any other Xapian database with suitable contents) via a web front-end
provided by omega, a CGI application.  A search can also be done over a
more than one database at once.

The CGI program's parameters are documented in cgiparams.txt, and OmegaScript,
the language used to define its web interface, is documented in
omegascript.txt.  Omega ships with several OmegaScript templates and you can
use these, modify them, or just write your own.  See the "Supplied Templates"
section below for details of the supplied templates.

Omega parses queries using the Xapian::QueryParser class - for the supported
syntax, see queryparser.html in the xapian-core documentation
- available online at: http://www.xapian.org/docs/queryparser.html


Term construction
=================

Documents within an omega database are stored with two types of terms:
those used for probabilistic searching (the CGI parameter 'P'), and
those used for boolean filtering (the CGI parameter 'B'). Boolean
terms start with an initial capital letter denoting the 'group' of the
term (eg: 'M' for MIME type), while probabilistic terms are all
lower-case, and are also stemmed before adding to the
database.

The "english" stemmer is used by default - you can configure this for omindex
and scriptindex with "--stemmer LANGUAGE" (use 'none' to disable stemming, see
omindex --help for the list of accepted language names).  At search time you
can configure the stemmer by adding $set{stemmer,LANGUAGE} to the top of you
OmegaScript template.

The two term types are used as follows when building the query:
B(oolean) terms with the same prefix are ORed together, with all the
different prefix groups being ANDed together. This is then FILTERed
against the P(robabilistic) terms. This will look something like:

		      [ FILTER ]
		       /      \
		      /        \
		 P-terms      [     AND     ]
			       /     | ... \
                              /
			[    OR    ]
		       /      | ... \
                    B(F,1) B(F,2)...B(F,n)

Where B(F,1) is the first boolean term with prefix F, and so on.

The intent here is to allow filtering on arbitrary (and, typically,
orthogonal) characteristics of the document. For instance, by adding
boolean terms "Ttext/html", "Ttext/plain" and "P/press" you would be
filtering the probabilistic search for only documents that are both in
the "/press" site _and_ which are either of MIME type text/html or
text/plain. (See below for more information about sites.)

If there is no probabilistic query, the boolean filter is promoted to
be the query, and the weighting scheme is set to boolean.  This has
the effect of applying the boolean filter to the whole database.

In order to add more boolean prefixes, you will need to alter the
index_file() function in omindex.cc. omindex currently adds several
useful ones, detailed below.

Probabilistic terms are constructed from the title, body and keywords
of a document. (Not all document types support all three areas of
text.) Title terms are stored with position data starting at 0, body
terms starting 100 beyond title terms, and keyword terms starting 100
beyond body terms. This allows queries using positional data without
causing false matches across the different types of term.


Sites
=====

Within a database, omega supports multiple sites. These are recorded
using boolean terms (see 'Term construction', above) to allow
filtering on them.

Sites work by having all documents within them having a common base
URL. For instance, you might have two sites, one for your press area
and one for your product descriptions:

	http://example.com/press/index.html
	http://example.com/press/bigrelease.html
	http://example.com/product/bigproduct.html
	http://example.com/product/littleproduct.html

You could index all documents within http://example.com/press/ using a
site of '/press', and all within http://example.com/product/ using
'/product'.

Sites are also useful because omindex indexes documents through the
file system, not by fetching from the web server. If you don't have a
URL to file system mapping which puts all documents under one
hierarchy, you'll need to index each separate section as a site.

An obvious example of this is the way that many web servers map URLs
of the form <http://example.com/~<username>/> to a directory within
that user's home directory (such as ~<username>/pub on a Unix
system). In this case, you can index each user's home page separately,
as a site of the form '/~<username>'. You can then use boolean
filters to allow people to search only a specific home page (or a
group of them), or omit such terms to search everyone's pages.

Note that the site specified when you index is used to build the
complete URL that the results page links to. Thus while sites will
typically want to be relative to the hostname part of the URL (eg
'/site' rather than 'http://example.com/site'), you can use them
to have a single search across several different hostnames. This will
still work if you actually store each distinct hostname in a different
database.


omindex operation
=================

omindex is fairly simple to use:

----------------------------------------------------------------------
Usage: omindex [OPTION] --db DATABASE 
        --url BASEURL [BASEDIRECTORY] DIRECTORY

Index static website data via the filesystem.
  -d, --duplicates      set duplicate handling ('ignore' or 'replace')
  -p, --preserve-nonduplicates
		        don't delete documents not updated during
			replace duplicate operation
  -D, --db              path to database to use
  -U, --url             base url DIRECTORY represents
  -M, --mime-type	additional MIME mapping ext:type
  -l, --depth-limit=LIMIT
                        set recursion limit (0 = unlimited)
  -f, --follow          follow symbolic links
      --overwrite       create the database anew (the default is
                        to update if the database already exists).
  -h, --help            display this help and exit
  -v, --version         output version and exit
----------------------------------------------------------------------

You _must_ specify the database to index into (it's created if it
doesn't exist, but relevant directories must exist - this includes the
base directory for Quartz) and the base URL (which is used as the
site, and can be relative to the hostname - starts '/' - or absolute -
starts with a scheme, eg 'http://morestuffhere...').

You also need to tell omindex which directory to index. This should be
either a single directory (in which case it is taken to be the
directory base of the entire site being indexed), or as two arguments,
the first being the directory base of the site being indexed, and the
second being a relative directory within that to index.

For instance, in the example above, if you separate your products by
size, you might end up with:

	http://example.com/press/index.html
	http://example.com/press/bigrelease.html
	http://example.com/product/large/bigproduct.html
	http://example.com/product/small/littleproduct.html

If the entire website is stored in the file system under the directory
/www/example, then you would probably index the site in two
passes, one for the '/press' site and one for the '/product' site. You
might use the following commands:

$ omindex -p --db /var/lib/omega/data/default --url /press /www/example/press
$ omindex -p --db /var/lib/omega/data/default --url /product /www/example/product

If you add a new large product, but don't want to reindex the whole of
the product section, you could do:

$ omindex -p --db /var/lib/omega/data/default --url /product /www/example/product large

and just the large products will be reindexed. You need to do it like that, and
not as:

$ omindex -p --db /var/lib/omega/data/default --url /product/large /www/example/product/large

because that would make the large products part of a new site,
'/product/large', which is unlikely to be what you want, as large
products would no longer come up in a search of the product
site. (Note that the --depth-limit option may come in handy if you have
sites '/product' and '/product/large', or similar.)

Currently omindex can index:

* HTML (.html, .htm, .shtml)
* PHP (.php) - our HTML parser knows to ignore PHP code
* text files (.txt, .text)
* PDF (.pdf) if pdftotext is available (comes with xpdf)
* PostScript (.ps, .eps, .ai) if ps2pdf (from ghostscript) and pdftotext (comes
  with xpdf) are available
* OpenOffice/StarOffice documents (.sxc, .stc, .sxd, .std, .sxi, .sti, .sxm,
  .sxw, .sxg, .stw) if unzip is available
* OpenDocument format documents (.odt, .ods, .odp, .odg, .odc, .odf, .odb,
  .odi, .odm, .ott, .ots, .otp, .otg, .otc, .otf, .oti, .oth) if unzip is
  available
* MS Word documents (.doc, .dot) if antiword is available
* MS Excel documents (.xls, .xlb, .xlt) if xls2csv is available (comes with catdoc)
* MS Powerpoint documents (.ppt, .pps) if catppt is available (comes with catdoc)
* Wordperfect documents (.wpd) if wpd2text is available (comes with libwpd)
* MS Works documents (.wps, .wpt) if wps2text is available (comes with libwps)
* AbiWord documents (.abw)
* Compressed AbiWord documents (.zabw) if gzip is available
* Rich Text Format documents (.rtf) if unrtf is available
* Perl POD documentation (.pl, .pm, .pod) if pod2text is available
* TeX DVI files (.dvi) if catdvi is available

If you have additional extensions that represent one of these types, you need
to add an additional MIME mapping using the --mime-type option. For instance:

$ omindex --db /var/lib/omega/data/default --url /press /www/example/press \
          --mime-type doc:application/postscript

The syntax of --mime-type is 'ext:type', where ext is the extension of
a file of that type (everything after the last '.'), and type is one
of:

    text/html
    text/plain
    text/rtf
    text/x-perl
    application/msword
    application/pdf
    application/postscript
    application/vnd.ms-excel
    application/vnd.ms-powerpoint
    application/vnd.ms-works
    application/vnd.oasis.opendocument.text
    application/vnd.oasis.opendocument.spreadsheet
    application/vnd.oasis.opendocument.presentation
    application/vnd.oasis.opendocument.graphics
    application/vnd.oasis.opendocument.chart
    application/vnd.oasis.opendocument.formula
    application/vnd.oasis.opendocument.database
    application/vnd.oasis.opendocument.image
    application/vnd.oasis.opendocument.text-master
    application/vnd.oasis.opendocument.text-template
    application/vnd.oasis.opendocument.spreadsheet-template
    application/vnd.oasis.opendocument.presentation-template
    application/vnd.oasis.opendocument.graphics-template
    application/vnd.oasis.opendocument.chart-template
    application/vnd.oasis.opendocument.formula-template
    application/vnd.oasis.opendocument.image-template
    application/vnd.oasis.opendocument.text-web
    application/vnd.sun.xml.calc
    application/vnd.sun.xml.calc.template
    application/vnd.sun.xml.draw
    application/vnd.sun.xml.draw.template
    application/vnd.sun.xml.impress
    application/vnd.sun.xml.impress.template
    application/vnd.sun.xml.math
    application/vnd.sun.xml.writer
    application/vnd.sun.xml.writer.global
    application/vnd.sun.xml.writer.template
    application/vnd.wordperfect
    application/x-abiword
    application/x-abiword-compressed
    application/x-dvi

If you wish to remove a MIME mapping, you can do this by omitting the type -
for example to not index .doc files, use: --mime-type doc:

--duplicates configures how omindex handles duplicates (detected on
URL). 'ignore' means to ignore a document if it already appears to be
in the database; 'replace' means to replace the document in the
database with a new one by indexing this file, and 'duplicate' means
to index this file as a new document, leaving the previous one in the
database as well. The last strategy is very fast, but is liable to do
strange things to your results set. In general, 'ignore' is useful for
completely static documents (eg: archive sites), while 'replace' is
the most generally useful.

With 'replace', omindex will remove any document it finds in the
database that it did not update - in other words, it will clear out
everything that doesn't exist any more. However if you are building up
an omega database with several runs of omindex, this is not
appropriate (as each run would delete the data from the previous run),
so you should use the --preserve-nonduplicates. Note that if you
choose to work like this, it is impossible to prune old documents from
the database using omindex. If this is a problem for you, an
alternative is to index each subsite into a different database, and
merge all the databases together when searching.

--depth-limit allows you to prevent omindex from descending more than
a certain number of directories.  If you wish to replicate the old
--no-recurse option, use ----depth-limit=1.


Boolean terms
=============

omindex will create the following boolean terms when it indexes a
document:

T	MIME type
H	hostname of site (if supplied - this term won't exist if you
	index a site with base URL '/press', for instance)
P	path of site (ie: the rest of the site base URL)
U	full URL of indexed document - if the resulting term would be > 240
	characters, a hashing scheme is used to prevent omindex overflowing
	the Xapian term length limit.



D	date (numeric format: YYYYMMDD)
	date can also have the magical form "latest" - a document indexed
	by the term Dlatest matches any date-range without an end date.
	You can index dynamic documents which are always up to date
	with Dlatest and they'll match as expected.  (If you use sort by date,
	you'll probably also want to set the value containing the timestamp to
	a "max" value so dynamic documents match a date in the far future).
W	"weak" (approximately 10 day intervals, taken as YYYYMMD from
	the D term, and changing the last digit to a '2' if it's a '3')
M	month (numeric format: YYYYMM)
Y	year (four digits)

omega configuration
===================

Most of the omega CGI configuration is dynamic, by setting CGI
parameters. However some things must be configured using a
configuration file.  The configuration file is searched for in
various locations:
 - Firstly, if the "OMEGA_CONFIG_FILE" environment variable is
   set, its value is used as the full path to a configuration file
   to read.
 - Next (if the environment variable is not set, or the file pointed
   to is not present), the file "omega.conf" in the same directory as
   the Omega CGI is used.
 - Next (if neither of the previous steps found a file), the file
   "${sysconfdir}/omega.conf" (eg, /etc/omega.conf on Linux systems)
   is used.
 - Finally, if no configuration file is found, default values are used.

The format of the file is very simple: a line per option, with the
option name followed by its value, separated by a whitespace.  Blank
lines are ignored.  If the first non-whitespace character on a line
is a '#', omega treats the line as a comment and ignores it.

The current options are 'database_dir' (the directory containing all the
Omega databases), 'template_dir' (the directory containing the OmegaScript
templates), and 'log_dir' (the directory which the omegascript $log command
writes log files to).

The default values (used if no configuration file is found) are:
----------------------------------------------------------------------
database_dir /var/lib/omega/data
template_dir /var/lib/omega/templates
log_dir /var/log/omega
----------------------------------------------------------------------

Note that, with apache, environment variables may be set using mod_env, and
with apache 1.3.7 or later this may be used inside a .htaccess file.  This
makes it reasonably easy to share a single system installed copy of Omega
between multiple users.


Supplied Templates
==================


The OmegaScript templates supplied with Omega are:

 * query - This is the default template, providing a typical Web search
   interface.
 * topterms - This is just like query, but provides a "top terms" feature
   which suggests terms the user might want to add to their query to
   obtain better results.
 * godmode - Allows you to inspect a database showing which terms index
   each document, and which documents are indexed by each term.
 * opensearch - Provides results in OpenSearch format (for more details
   see http://www.opensearch.org/).
 * xml - Provides results in a custom XML format.

There are also "helper fragments" used by the templates above:

 * inc/anyalldropbox - Provides a choice of matching "any" or "all" terms
   by default as a drop down box.
 * inc/anyallradio - Provides a choice of matching "any" or "all" terms
   by default as radio buttons.
 * toptermsjs - Provides some Javascript used by the topterms template.


Document data construction
==========================

This is only useful if you need to inject your own documents into the
database independently of omindex, such as if you are indexing
dynamically-generated documents that are served using a server-side
system such as PHP or ASP, but which you can determine the contents of
in some way, such as documents generated from reasonably static
database contents.

The document data field stores some summary information about the
document, in the following (sample) format:

----------------------------------------------------------------------
url=<baseurl>
sample=<sample>
caption=<title>
type=<mimetype>
----------------------------------------------------------------------

Further fields may be added (although omindex doesn't currently add any
others), and may be looked up from OmegaScript using the $field{}
command.

As of Omega 0.9.3, you can alternatively add something like this near the
start of your OmegaScript template:

$set{fieldnames,$split{caption sample url}}

Then you need only give the field values in the document data, which can
save a lot of space in a large database.  With the setting of fieldnames
above, the first line of document data can be accessed with $field{caption},
the second with $field{sample}, and the third with $field{url}.
