gecko-dev/webtools/bonsai/INSTALL

# -*- mode: indented-text -*-
#
# Author: Artem Belevich <abelevic@ctron.com> 
#
# (Changes have been made to Artem's original doc, as things evolve.)
#
#

**********************************************************************

As it's said in README "This is not very well packaged code.  It's
not packaged at all.  Don't come here expecting something you plop in
a directory, twiddle a few things, and you're off and using it.  Much
work has to be done to get there." 

This file is intended to make some things *easier* but not easy. You
are still required to make some changes on your own. There is no
guaranteed solution yet and it's unlikely that there will be one in
the nearest future.

**********************************************************************


0. OVERVIEW

Some time ago I've seen Linux Source Navigator (LSN) at
http://sunsite.unc.edu/linux-source. I was impressed. 
It was and is a wanderful tool to explore Linux kernel source code.

Then Mozilla.org came up with a more elaborate tool that includes
source browser with crossreferencing (LXR http://lxr.linux.no) and CVS
tree control (Bonsai - http://www.mozilla.org/bonsai.html). 
While LXR formatting is not as pretty as LSN's one, it has a huge
advantage - it lets you see where the identifier is defined and used.
And Bonsai brings nice and easy (though sometimes incompatible with
browsers other but Netscape's own) interface to the CVS history. This
includes getting list of changes, giffs between revisions, etc.

All in all LXR+Bonsai+other stuff beneath is a useful tool capable
of handling huge projects. 

It's not that easy to make it work with other source tree but
Mozilla's own but it's possible. And there are a lot of things to
improve. Now I'm going to concentrate on the first goal - to make it
work. 


1. GETTING IT UP

   First of all you have to get all the tools in mozilla's
mozilla/webtools CVS repository. This includes lxr,bonsai,registry
and tinderbox. You're likely will not need neither tinderbox nor
registry but get them just in case.

   To get the sources you have to follow instructions on
http://www.mozilla.org/bonsai.html.

   OK, now you've got the sources but don't rush to try it right
away. It's likely that you will not be able to even start most of
the scripts. There are more things you will have to get and install.
The short list of the things you will need:

   1) MySQL database server.
   2) Tcl 7.6
   3) TclX 7.6
   4) mysqltcl program (hmm.. This was tricky.. Read on)
   5) Perl 5.004+ with Mysql module (included with MySQL).
   6) Date::Parse module for Perl
   7) Some kind of HTTP server so you could use CGI scripts

   You could try running the ./configure script to see what tools it
complains about right now. Mind you, it won't check for the MySQL
database.

1.1 Getting and setting up MySQL database

   Visit MySQL homepage at http://www.tcx.se and grab the latest
stable binary release of the server. Sure, you can get sources and
compile them yourself, but binaries are the easiest and the fastest
way to get it up and running. Follow instructions found in
manual. There is a section about installing binary-only
distributions. 

   You should create database bonsai. It may be a good idea to make it
writable by all users on your machine and change access level
later. This would save you a lot of time trying to guess whether it's
permissions or a mistake in the script that make things fail.

1.2-3 Getting and building Tcl & TclX 8.0.4

    Bonsai works with Tcl & TclX 7.6 too, but if you need to build,
you might as well get the latest stable versions.
Tcl homepage is at http://www.scriptics.com. You may get sources
for UNIX from ftp://ftp.scriptics.com/pub/tcl/tcl8_0/tcl8.0.4.tar.gz.
TclX is an extension for Tcl that adds a lot of useful functions that
are heavily used in Bonsai tool. 
TclX page is http://www.neosoft.com/tclx. Download sources from
ftp://ftp.neosoft.com/pub/tcl/TclX/tclX8.0.4.tar.gz. Watch out for the
case of the letters in URL. These guys are going to bring some fun
into your life by spelling their program name in various ways.

    Now you've probably got both Tcl and TclX 8.0.4. You may try to use
later versions but I'm not sure about results. Unfortunately I'm not an
expert in "Tcl&Co.".

    Build and install Tcl first. Then build and install TclX. This
should go without serious problems

1.4 mysqltcl - the tricky part

   Grab mysqltcl 1.53 from MySQL site's contributed software area
(http://www.tcx.se/Contrib/) I've used version 1.53 and it works for
me, though you may try more recent version at your own risk. You're
risking anyway.

   You will need to patch mysqltcl to include support for TclX. A
patch is provided in Appendix A.

1.5 Perl + Mysql

   Besides mysqltcl you will need Perl 5.004 with DB and Mysql
extensions. 

   DB is required to use LXR browser and crossreferencer for storing
its database. Mysql is used by Bonsai.

   If you have Perl already installed, try to run genxref program from
LXR suite. If it complains that it misses DB terribly then you're
probably will have to get and install DB 1.86 distribution from one of the
CPAN (www.cpan.org) mirrors in src/misc directory. I personally got it
from http://www.cpan.org/src/misc/db.1.86.tar.gz. Having DB compiled
and installed you will also have to rebuild and reinstall Perl
itself so It would recognize and compile DB module in. This can be
tricky if you have DB installed in some strange place as I did. 
I've got an error during linking phase - there was a function missing
in hash/ndbm.c file, so I just commented it out. It may potentially
cause troubles, but I think it does not matter in our case as this
was intended only for DBM compatibility - the feature we don't really
use. 

   Now you hopefully have Perl + DB compiled installed and working.
Time to set up Mysql module. This one is easy. Just follow
instructions in MySQL manual. You have to read manuals sometimes.. 
I think I'm getting older.. 8-)

   Next step is to get TimeDate module from one of the CPAN mirrors.
Go to CPAN search page
(http://theory.uwinnipeg.ca/search/cpan-search.html) and search for
the "TimeDate" module. Then get it and install.

1.6 HTTP server

You have a freedom of choice here - Apache, Netscape or any other
server on UNIX would do. The only thing - to make configuration easier
you'd better run HTTP daemon on the same machine that you run MySQL
server on. Make sure that you can access 'bonsai' database with user
id you're running the daemon with.


2. TWEAKING THE TOOLS

   Now you should have all necessary tools to be able to run LXR and
Bonsai scripts and see why the wouldn't work for you right now.

2.1 LXR

   The first thing to set up is LXR tool. All it needs is the source
tree (not CVS tree). It's relatively easy and works almost right of
the box. Follow instructions in LXR README file. 

   Having set LXR you will see that regardless what your source tree
contains you will see that everything refers to it as Mozilla. Mozilla
is a great thing and this tool was primarily tailored to mozilla tree
but you'd like to control your own tree. First step is to edit your

Here is the short list of changes I had to make

  file: ident
    1) change "&root=/cvsroot" to tour CVSROOT path
    2) change "file=/mozilla/" to the directory under CVSROOT where
       your sources are. In my case it is just "/"

  file: index.html
    Nothing vital here but probably worth changing to reflect your own
    environment

  file: lxr.conf
    Changes to this file are described in LXR README file and are
    quite simple.

  file: source
    You may find it useful to uncomment "$img = "/icons/..." lines if
    you yse Explorer as it does not have internal-gopher-* images
    built in. Actually Bonsai contains a lot of netscapism that will
    make your IE4 unhappy anyway. You'd better stick with Netscape if
    you are going to use LXR/Bonsai

  file: template-*
    Here you will probably want to watch closely at the places whare
    you see the word 'mozilla' near '.cgi'. There are a lot of
    mozilla-specific paths hardcoded
    
    cgange/get rid of banner that loads straight from mozilla.org that
    may be very dangerous if you're working for micro$oft and your
    boss comes by.. 8-) 

2.2 Bonsai

   This stuff sometimes gets very specific about your CVS repository
setup. You have to make a lot of changes until more portable
configuration mechanism is introduced. 

   These steps should create a basic Bonsai install:

   ./configure
   make install

   You might want to give the option --prefix=<path> to configure to
install Bonsai in another place than /usr/local, e.g. /var/www.  It
will make a new directory named "bonsai" in the prefix directory you specify.

   Edit data/configdata file as described in README file. Create
appropriate data/XXX directory for each tree XXX you've configured
in 'configdata'.

   Go to the data directory and run 

   trapdoor <admin password here> >data/passwd

   it will set up admin's password.

   file: cvsblame.cgi
      @line 202: you may add your own translation to help find
      appropriate RCS file for the given source file.
      TODO: We'd better get this information from CVS/Repository
            file. It's more generic than hardcoded tweaks.

   file: cvslog.cgi
      @line 154: the same tweaks to find RCS file for given source file
      @ line 475: add translations to $lxr_path to help find source
		  file for given RCS file.
      TODO: This should be done using information in
            $CVSROOT/CVSROOT/modules file. Typical example is the
	    following line: 'mymodule -d mydir repositorydir' which
	    causes files for module mymodule from repositorydir to be
	    placed in mydir. This contains enough information about
	    backward translation mydir/file->repositorydir/file,v


   file: cvsquery.cgi
      @line 25: There are some cases when cvsroot is not passed as
          parameter for cvsquery.cgi. You should better check for
	  empty CVS_ROOT and set it to your CVSROOT path.

That's basically it. With some luck and persistence you will have 90%
working system at this point. A lot of these things are just asking to be
fixed in near feature. And I hope they will be.

3. Setting up database

   This is quite simple but time consuming operation.
First create database structure using maketable.sh script. You might
want to edit it to use the user and password you want for the bonsai
database.
Then go to Bonsai administration page and press "Rebuild CVS history"
button. Then you may go to the theater and watch a movie or two. It
will talke a lot of time. It takes several seconds to process one
file. The more revisions in file the more time it will take. My SUN
workstation with 2x200Mhz UltraSPARC processors run about an hour to
process about 4K files with 20K+ revisions. Your mileage may vary.

Copy "dolog.pl" to your CVSROOT directory, and check it in. Add
"dolog.pl" to CVSROOT/checkoutlist, and check it in. Then, add a line
to your CVSROOT/loginfo file that says something like:

      ALL      $CVSROOT/CVSROOT/dolog.pl -r /cvsroot bonsai-checkin-daemon@my.bonsai.machine

Replace "/cvsroot" with the name of the CVS root directory, and
"my.bonsai.machine" with the name of the machine Bonsai runs on. Now,
on my.bonsai.machine, add a mail alias so that mail sent to
"bonsai-checkin-daemon" will get piped to handleCheckinMail.tcl. The
first argument to handleCheckinMail.tcl is the directory that bonsai
is installed in. E.g. in /etc/aliases, add

 bonsai-checkin-daemon: "|/usr/local/bonsai/handleCheckinMail.tcl /usr/local/bonsai"

or whatever is appropriate for your mail transport agent.
 
4. Things to do

  a) There should be better way to track CVS tree changes. Now it's done
by making CVS send e-mail about each checkin.  (See the comments at
the top of dolog.pl for some clues.) One alternative theory would be
to take advantage of the CVS history command, which provides
all necessary information to get list of files recently commited, so
there is no need to send/process email. Just set up a cron job that
will periodically look for CVS tree changes and update database.  On
the other hand, it's not at all clear how efficient the cvs history
command is for large, active repositories.

  b) Better configuration. One should not hardcode CVS tree <-> Source
tree  translations. Another thing to configure - banners.

  c) LXR could be improved in a number of ways. Using MySQL database
instead of DB would probably be a good idea. It's unclear what impact
it will have on performance though. Incremental database updates would
be nice. It might be also nice to borrow syntax highliting from LSN.


5. Conclusion.

   OK. This may or may not work for you. But I hope you had a great
time trying. Or just reading.

   Any suggestions/additions are welcome.


*******************************************************
APPENDIXES
*******************************************************

APPENDIX 1. Patch to build mysqltcl with TclX
---------------------------------------------

diff -u -r mysqltcl-1.53/Makefile mysqltcl-1.53p/Makefile
--- mysqltcl-1.53/Makefile	Mon Jul  6 18:11:55 1998
+++ mysqltcl-1.53p/Makefile	Wed Apr 28 09:48:55 1999
@@ -10,7 +10,7 @@
 TKHOME   = /usr
 MYSQLHOME = /usr/local/mysql
 
-SHARED = yes
+SHARED = no
 #PLATFORM=SunOS-5.5.1-sparc
 #PLATFORM=SunOS-5.6-sparc
 PLATFORM=Linux-2.0-i586
@@ -59,7 +59,7 @@
 CPPFLAGS  = -I$(XHOME)/include -I$(TCLHOME)/include -I$(TKHOME)/include \
 	-I$(MYSQLHOME)/include
 LOADLIBES = -L$(XHOME)/lib -L$(TCLHOME)/lib -L$(TKHOME)/lib \
-	-L$(MYSQLHOME)/lib -lmysqlclient -ltk8.0 -ltcl8.0 -lX11 -lm -ldl $(LIBS) 
+	-L$(MYSQLHOME)/lib -lmysqlclient -ltclx8.0.4 -ltk8.0 -ltcl8.0 -lX11 -lm -ldl $(LIBS) 
 
 all: $(OUTPUT)
 
diff -u -r mysqltcl-1.53/mysqltcl.c mysqltcl-1.53p/mysqltcl.c
--- mysqltcl-1.53/mysqltcl.c	Mon Jul  6 17:35:17 1998
+++ mysqltcl-1.53p/mysqltcl.c	Mon Apr 26 11:21:10 1999
@@ -27,7 +27,7 @@
  * CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
  */
 
-#include <tcl.h>
+#include <tclExtend.h>
 #include <mysql.h>
 
 #include <errno.h>
diff -u -r mysqltcl-1.53/mysqltclsh.c mysqltcl-1.53p/mysqltclsh.c
--- mysqltcl-1.53/mysqltclsh.c	Wed Mar 18 13:44:55 1998
+++ mysqltcl-1.53p/mysqltclsh.c	Mon Apr 26 11:26:26 1999
@@ -17,7 +17,7 @@
 #include <X11/Intrinsic.h>
 #endif
 
-#include "tcl.h"
+#include "tclExtend.h"
 
 /*
  * The following variable is a special hack that is needed in order for
@@ -61,7 +61,7 @@
 #ifdef TCL_XT_TEST
     XtToolkitInitialize();
 #endif
-    Tcl_Main(argc, argv, Tcl_AppInit);
+    TclX_Main(argc, argv, Tcl_AppInit);
     return 0;			/* Needed only to prevent compiler warning. */
 }
 
@@ -92,6 +92,10 @@
 	return TCL_ERROR;
     }
 
+       if (Tclx_Init(interp) == TCL_ERROR) {
+	return TCL_ERROR;
+    }
+
 #ifdef TCL_TEST
 #ifdef TCL_XT_TEST
      if (Tclxttest_Init(interp) == TCL_ERROR) {
@@ -135,6 +139,6 @@
      * then no user-specific startup file will be run under any conditions.
      */
 
-    Tcl_SetVar(interp, "tcl_rcFileName", "~/.tclshrc", TCL_GLOBAL_ONLY);
+/*    Tcl_SetVar(interp, "tcl_rcFileName", "~/.tclshrc", TCL_GLOBAL_ONLY);*/
     return TCL_OK;
 }