Discussion:
[Axiom-developer] [build-improvements] Requests for discussion
Gabriel Dos Reis
2006-08-01 19:50:23 UTC
Hi,

Instead of fiddling at infinitum with the new build machinery on my
local disk. I decided to trim it down to something "simple" to
request feedback about some aspects of the work.

The core goal is to move Axiom's build system to GNU Autotools which
have become quite standard now, in the Unix-world.

For that I've converted the current configure file to configure.ac
(which Autoconf will use to regenerate configure). The main
motivation here is factor out many platforms variability issues and
dump them on Autoconf -- who already knows how to solve them.

Axiom has the requirement that the source files must be literate --
use of pamphlet files. Autoconf expects its input to be in particular
form. Furthermore, the pamphlet file requirement introduces an
unpleasant bootstrapping issue that will hopefully be resolved.
So, in its current form, I put the literate stuff in comments, so that
if you do

cat configure.ac | sed -e 's/^## //' > configure.tex

you can an almost LaTeX file. "Almost" because LaTeX commands need to
start with a backslash. That character used to be reserved for line
continuation in shell world. So to avoid any surprise, I used the
percent character (which unfortunately is LaTeX comment character). So
to get the LaTeX document, one needs to apply another sed that
replaces the percent character with the backslash character.

I've tested that I can feed the file to Autoconf, get a configure file
out of it and launch a successful build.

This work is being done on the "build-improvements" branch -- and it
has not been committed there yet.

Before painting myself into a corner, I would like to have feedbacks
from all y'all.

Thanks.

-- Gaby

## %documentclass[12pt]{article}
## %usepackage{axiom}
## %usepackage[latin1]{inputenc}
## %usepackage[T1]{fontenc}
## %usepackage{fancyvrb}
## %usepackage{pslatex}
## %usepackage{url}

## %newcommand{%file}[1]{%textsf{#1}}
## %newcommand{%code}[1]{%texttt{#1}}
## %newcommand{%email}[1]{%url{#1}}
## %CustomVerbatimEnvironment{chunk}{Verbatim}{frame=none,fontsize=%small}

## %title{The Toplevel %file{configure.ac} File}
## %author{Gabriel Dos~Reis}
##
## %begin{document}
## %maketitle
##
## %begin{abstract}
## ...
## %end{abstract}

## %section{Introduction}
## This is the top-level Autoconf file that sets up the minimum build
## environment for Axiom. At the moment, the mainline version
## of Axiom is driven by %file{Makefile} pamphlet files. This effort
## strives to move the build machinery to more abstract description
## and conventional ones.
## The task is compounded by the fact that the Axiom system is very
## complex -- but much less complex, I suspect, than say GCC. There does not
## seem to be good reasons why Axiom should build its own ghetto.
##
## Autoconf supports two kinds of comments:
## %begin{enumerate}
## %item %code{dnl} style, and
## %item %code{%#} style.
## %end{enumerate}
## Comments introduced with %code{dnl} do not appear in the %file{configure}
## output file. Comments starting with %code{%#} appear verbatim in the
## %file{configure} generated file.
##
## I have been trying to write this Autoconf file so that it can
## eventually be processed by both Autoconf and noweb -- the currently en
## vogue literate programming tool used by Axiom. To accommodate that
## requirement, I've strived for introducing noweb chunkname in
## %code{dnl}-style comments, whereas text intended to be part of the text
## processed by LaTeX is introduced by %verb!## ! This is so that the
## poor masochist who will be debugging the generated %file{configure} file
## (if he/she so elected) has a clue about where the macros are expanded
## from and what their purposes are.

## %section{Old Story}
##
## The contents of top-level %file{configure} file from mainline is
## reproduced below. It will serve as the initial basis for the new,
## improved, build machinery.

## %begin{verbatim}
## # The sysname function uses uname -s to try to decode what kind of
## # system to build. Currently the return value of uname is mapped as
## # Linux --> linux
## # MINGW32_NT-5.1 --> windows
## # SunOS --> Solaris9
## # Fedora Core 3 --> fedora3
## # freebsd --> freebsd
## #
## # The solaris platform needs patch->gpatch, awk->gawk, tar->gtar
##
## sysname () {
## if [ -f /etc/redhat-release ] ;
## then
## SYSNAME=cat /etc/redhat-release
## if [ "$SYSNAME" = "Fedora Core release 3 (Heidelberg)" ] ; ## then SYSNAME=fedora3 ## fi ## echo SYSNAME=$SYSNAME
## fi
## if [ ! "$SYSNAME" = "fedora3" ] ; ## then ## SYSNAME=uname -s ## echo$SYSNAME
## if [ "$SYSNAME" = "Linux" ] ; then SYSNAME=linux ## elif [ "$SYSNAME" = "MINGW32_NT-5.1" ] ; then SYSNAME=windows
## elif [ "$SYSNAME" = "SunOS" ] ; then SYSNAME=solaris9 ## elif [ "$SYSNAME" = "freebsd" ] ; then SYSNAME=freebsd
## else
## echo Your system name is $SYSNAME ## echo We do not know how to build for this kind of system ## echo Send a note to axiom-***@nongnu.org about it ## echo ## exit 0 ## fi ## fi ## ## } ## ## # This function checks for the gawk command. ## # If it exists then AWKNAME is the complete pathname ## ## checkgawk() { ## AWKNAME=which gawk 2>>trace ## if [ -n "$AWKNAME" ] ; then
## if [ -x $AWKNAME ] ; then ## echo ## fi ## fi ## } ## ## # This function checks for the nawk command. ## # If it exists then AWKNAME is the complete pathname ## ## checknawk() { ## AWKNAME=which nawk 2>>trace ## if [ -n "$AWKNAME" ] ; then
## if [ -x $AWKNAME ] ; then ## echo ## fi ## fi ## } ## ## # This function checks for the awk command. ## # If it exists then AWKNAME is the complete pathname ## ## checkawk() { ## AWKNAME=which awk 2>>trace ## if [ -n "$AWKNAME" ] ; then
## if [ -x $AWKNAME ] ; then ## echo ## fi ## fi ## } ## ## # This function uses the check*awk functions to decide ## # whether the system can build noweb. If one of gawk, nawk or awk ## # are not found we fail. ## needAwk () ## { ## checkgawk ## if [ -z "$AWKNAME" ] ; then
## checknawk
## if [ -z "$AWKNAME" ] ; then ## checkawk ## if [ -z "$AWKNAME" ] ; then
## echo We need the commands gawk, nawk, or awk
## exit 0
## fi
## fi
## fi
## }
##
## # The mustSet function tells the user what needs to be typed on the
## # command line. If any extra variables need to be set we add them here.
## # Currently the only thing we check if for the presence of gawk, which
## # is the default in the Makefile. If gawk does not exist we can use
## # either nawk or awk but the user has to specify that on the command line.
##
## # We check the system we are using with the uname command and try to
## # generate the appropriate value. We fail otherwise.
##
## # We generate the appropriate command line that the user should use.
##
## mustSet() {
## echo
## echo ===================================================
## echo You must set your AXIOM and PATH variables. Type:
## echo
## echo To build the rest of the system type:
## echo
## echo export AXIOM=pwd/mnt/$SYSNAME ## echo 'export PATH=$AXIOM/bin:$PATH' ## if [ "$SYSNAME" = "freebsd" ] ; then
## echo Note that freebsd usually has noweb available
## echo If you wish to use the standard version you must type
## echo touch noweb
## echo If you wish to use a pre-installed GCL you must type
## echo make GCLVERSION=gcl-system
## fi
## if [ "$SYSNAME" = "solaris9" ] ; ## then echo make AWK=gawk TAR=gtar PATCH=gpatch ## elif [ "basename$AWKNAME" = "gawk" ] ;
## then echo make
## else echo make AWK=$AWKNAME ## fi ## echo ## echo configure finished. ## } ## ## ######################################################################### ## # This is the main line of configure logic. ## # (1) We test to see if we understand this system name. So far ## # the recognized strings from uname -s are translated as: ## # Linux --> linux ## # MINGW32_NT-5.1 --> windows ## # SunOS --> Solaris9 ## # Fedora Core 3 --> fedora3 ## # freebsd --> freebsd ## # (1) We test for the AWK variable. We need one of gawk, nawk, or awk ## # in order to build the noweb software. ## # (2) Then we output the final message for the user. ## # ## # The solaris platform needs patch->gpatch, awk->gawk, tar->gtar ## ######################################################################### ## ## sysname ## needAwk ## ## if [ "x$AXIOM" = "x" ] ;
## then mustSet
## else
## if [ ! "dirname $AXIOM" = "pwd/mnt" ] ## then mustSet ## else ## echo Configure complete. Now type ## echo ## echo make ## echo ## fi ## fi ## %end{verbatim} ## %section{Basic Setup} ## %subsection{Autoconf Initialization} ## ## The Autoconf machinery needs to be initialized with several pieces of ## information: ## %begin{itemize} ## %item the %emph{name} of the system --- Axiom silver branch'' ## %item its %emph{version}. I choose to use the date of last checkin. ## It should probably include the revision number so as to ## unambiguously identify which Axiom flavour du jour is being ## built; ## %item and where to send feedback, %emph{e.g.} bug reports. I have chosen ## the %email{axiom-developer} list. That could change in the future if ## we reach a high volume traffic. For the moment, we don't seem to ## suffer from traffic... ## %end{itemize} dnl %begin{chunk} AC_INIT([Axiom silver branch], [2006-07-15], [axiom-***@nongnu.org]) dnl %end{chunk} ## Autoconf needs some auxilary files that are present in the sub-directory ## %file{config}. Autoconf needs to be told about that. dnl %begin{chunk} AC_CONFIG_AUX_DIR(config) dnl %end{chunk} ## Notice that since we don't use Automake (yet), we don't initialize ## the Automake subsystem. ## We require Autoconf 2.59 or higher from the developer part. Please, ## note that this is no requirement on the user build environment. All, ## it means is that if someone makes changes to the current %file{configure.ac} ## file, that someone needs to have Autoconf 2.59 or higher to process this ## file in order to regenerate %file{configure}. dnl %begin{chunk} AC_PREREQ([2.59]) dnl %end{chunk} ## The Autoconf system implements a very basic, simple-minded, sanity check ## whereby it will refuse to run %file{configure} if the source tree does ## not contain a specified file, that serves a witness for a bona fide source ## tree. Here, I have chosen %file{Makefile.pamphlet} from the %file{src} ## subdirectory. dnl %begin{chunk} AC_CONFIG_SRCDIR(src/Makefile.pamphlet) dnl %end{chunk} ## %subsection{Build Environment} ## ## Standard build environments consist of ## %begin{enumerate} ## %item the %emph{host} platform, ## %item the %emph{build} platform, and ## %item the %emph{target} platform. ## %end{enumerate} ## FIXME: Example on these notions. ## ## Get the canonical names for the above three platforms. After call to ## this macro, those values are available in the variables %code{host}, ## %code{build}, and %code{target}, respectively. dnl %begin{chunk} AC_CANONICAL_SYSTEM dnl %end{chunk} ## For the moment, we don't support cross-compiling, nor Canadian cross. ## Consequently, we must bail out if the build is not native. dnl %begin{chunk} if test$host != $build -o$host != $target; then AC_MSG_ERROR([Sorry, only native builds are currently supported]) fi dnl %end{chunk} ## Check for a C compiler, %textsl{GCC/gcc} preferably. dnl %begin{chunk} AC_PROG_CC dnl %end{chunk} ## Check for a usable 'install' program. dnl %begin{chunk} AC_PROG_INSTALL dnl %end{chunk} ## The old build machinery needs 'awk'. Currently, it checks for ## 'gawk', 'nawk', and 'awk'. Autoconf has a predefined test for that ## task. It checks for 'gawk', 'mawk', 'nawk', and 'awk' in that order. ## That should be OK and match Axiom's need. ## The old build system claims that on solaris9, gawk, gtar ## and gpatch are required (with no much explanation of why). Notice ## that these programs are needed only to build Axiom; so we do ## check based on the value of %code{build}. dnl %begin{chunk} case${build} in
*-solaris9)
AC_CHECK_PROG([AWK], [gawk],
[gawk], [AC_MSG_ERROR([Axiom needs gawk])])

AC_CHECK_PROG([TAR], [gtar],
[gtar], [AC_MSG_ERROR([Axiom needs gtar])])

AC_CHECK_PROG([PATCH], [gpatch],
[gptach], [AC_MSG_ERROR([Axiom needs gpatch])])
;;

*)
AC_PROG_AWK

AC_CHECK_PROGS([TAR], [gtar tar],
[AC_MSG_ERROR([Axiom needs a tar program])])

AC_CHECK_PROGS([PATCH], [gpatch patch],
[AC_MSG_ERROR([Axiom needs a patch program])])
;;
esac
dnl %end{chunk}

## Once we have found those values, we must substitute them in the
## Makefiles. For the moment, these actions are effectless as the
## current makefile are hand-written and Autoconf-unaware.
dnl %begin{chunk}
AC_SUBST(AWK)
AC_SUBST(TAR)
AC_SUBST(PATCH)
dnl %end{chunk}

## Obviously we need the 'make' program. No build will proceed without
## it.
dnl %begin{chunk}
AC_CHECK_PROG([MAKE], [make],
[make], [AC_MSG_ERROR([make' program missing.])])

dnl %end{chunk}

## Here, we replicate the behaviour of the old configure, waiting for
## a better alternative, e.g. where it is not needed. THIS WILL BE
## REMOVED IN THE NEAR FUTURE.

## First, the old machinery has a very coarse target name "discovery"
dnl %begin{chunk}
if test -f /etc/redhat-release; then
SYSNAME=cat /etc/redhat-release
if test "$SYSNAME" = "Fedora Core release 3 (Heidelberg)"; then SYSNAME=fedora3 fi AC_MSG_NOTICE([SYSNAME=$SYSNAME])
fi
if test "$SYSNAME" != "fedora3"; then SYSNAME=uname -s AC_MSG_NOTICE([$SYSNAME])
case "$SYSNAME" in Linux) SYSNAME=linux ;; MINGW32_NT-5.1) SYSNAME=windows ;; SunOS) SYSNAME=solaris9 ;; freebsd) ;; *) AC_MSG_NOTICE([Your system name is$SYSNAME])
AC_MSG_NOTICE([We do not know how to build for this kind of system])
AC_MSG_ERROR([Send a note to axiom-***@nongnu.org about it])
esac
fi

must_set_AXIOM() {
AC_MSG_NOTICE([])
AC_MSG_NOTICE([===================================================])
AC_MSG_NOTICE([])
AC_MSG_NOTICE([You must set your AXIOM and PATH variables. Type:])
AC_MSG_NOTICE([])
AC_MSG_NOTICE([export AXIOM=pwd/mnt/$SYSNAME]) AC_MSG_NOTICE([export PATH=\$AXIOM/bin:\$PATH]) case "$SYSNAME" in
freebsd)
AC_MSG_NOTICE([Note that freebsd usually has noweb available])
AC_MSG_NOTICE([echo If you wish to use the standard version you must type])
AC_MSG_NOTICE([touch noweb])
AC_MSG_NOTICE([If you wish to use a pre-installed GCL you must type])
AC_MSG_NOTICE([make GCLVERSION=gcl-system])
;;
solaris9)
AC_MSG_NOTICE([make AWK=gawk TAR=gtar PATCH=gpatch])
;;
*)
AC_MSG_NOTICE([make AWK=$AWK]) esac } if test "x$AXIOM" = "x"; then
must_set_AXIOM
elif test "dirname $AXIOM" != "pwd/mnt"; then must_set_AXIOM else AC_MSG_NOTICE([configure complete. Now type ]) AC_MSG_NOTICE([ ]) AC_MSG_NOTICE([make]) AC_MSG_NOTICE([ ]) AC_MSG_NOTICE([export AXIOM=pwd/mnt/$SYSNAME])
AC_MSG_NOTICE([export PATH=\$AXIOM/bin:\$PATH])
fi
dnl %end{chunk}

## %end{document}
Ralf Hemmecke
2006-08-01 21:35:55 UTC
I don't see a ./config file in that directory.

Ralf

woodpecker:~/OTHER/Axiom/Silver/build-improvements>uname -a
Linux woodpecker 2.6.11-rc4 #3 Thu Feb 23 14:31:45 CET 2006 i686 GNU/Linux
woodpecker:~/OTHER/Axiom/Silver/build-improvements>autoconf configure.ac
Post by Gabriel Dos Reis
configure
woodpecker:~/OTHER/Axiom/Silver/build-improvements>./configure
configure: error: cannot find install-sh or install.sh in config ./config
Gabriel Dos Reis
2006-08-01 21:50:33 UTC
Ralf Hemmecke <***@hemmecke.de> writes:

| I don't see a ./config file in that directory.

Thanks for the feedback. Let me, first, merge your changes to silver,
then commit mine to build-improvements. I'll send you a ping for
testing.

Are you comfortable with the approach of my ad-hoc approach to
literacy here that is not mainstream Axiom not ALLPROSE? What would
you suggest?

-- Gaby
Gabriel Dos Reis
2006-08-02 09:22:39 UTC
Ralf Hemmecke <***@hemmecke.de> writes:

| I don't see a ./config file in that directory.

Ralf --

Based on the feedback I got, I implemented the Autoconf-based
configuration stuff as a pamphlet file (see below). At the moment,
the only thing it does is to mimic the existing (old) configure file.
Future work aims at reducing the build process to

./configure
make
make install

To test this, you don't need to run Autoconf. Just check out a
fresh copy of the build-improvements branch and proceed as usual.

Feedback appreciated.

Thanks!

-- Gaby

\documentclass[12pt]{article}
\usepackage{axiom}
\usepackage[latin1]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{fancyvrb}
\usepackage{pslatex}
\usepackage{noweb}
\usepackage{url}

\newcommand{\file}[1]{\textsf{#1}}
\newcommand{\email}[1]{\url{#1}}
\CustomVerbatimEnvironment{chunk}{Verbatim}{frame=none,fontsize=\small}

\title{The Toplevel \file{configure.ac} File}
\author{Gabriel Dos~Reis}

\begin{document}
\maketitle

\begin{abstract}
...
\end{abstract}

\section{Introduction}
This is the top-level Autoconf file that sets up the minimum build
environment for Axiom. At the moment, the mainline version
of Axiom is driven by \file{Makefile} pamphlet files. This effort
strives to move the build machinery to more abstract description
and conventional ones.
The task is compounded by the fact that the Axiom system is very
complex -- but much less complex, I suspect, than say GCC. There does not
seem to be good reasons why Axiom should build its own ghetto.

Autoconf supports two kinds of comments:
\begin{enumerate}
\item [[dnl]] style, and
\item [[#]] style.
\end{enumerate}
Comments introduced with [[dnl]] do not appear in the \file{configure}
output file. Comments starting with [[#]] appear verbatim in the
\file{configure} generated file. Because this source file is
literate, there almost never is a need to use the [[dnl]]-style
comment. Consequently, Autoconf comments in this file should be
of [[#]]-style. Such comments can be of value to the occasional
poor masochist who will be debugging the generated \file{configure}
file.

\section{Old Story}

The contents of top-level \file{configure} file from mainline is
reproduced below. It will serve as the initial basis for the new,
improved, build machinery.

\begin{verbatim}
# The sysname function uses uname -s to try to decode what kind of
# system to build. Currently the return value of uname is mapped as
# Linux --> linux
# MINGW32_NT-5.1 --> windows
# SunOS --> Solaris9
# Fedora Core 3 --> fedora3
# freebsd --> freebsd
#
# The solaris platform needs patch->gpatch, awk->gawk, tar->gtar

sysname () {
if [ -f /etc/redhat-release ] ;
then
SYSNAME=cat /etc/redhat-release
if [ "$SYSNAME" = "Fedora Core release 3 (Heidelberg)" ] ; then SYSNAME=fedora3 fi echo SYSNAME=$SYSNAME
fi
if [ ! "$SYSNAME" = "fedora3" ] ; then SYSNAME=uname -s echo$SYSNAME
if [ "$SYSNAME" = "Linux" ] ; then SYSNAME=linux elif [ "$SYSNAME" = "MINGW32_NT-5.1" ] ; then SYSNAME=windows
elif [ "$SYSNAME" = "SunOS" ] ; then SYSNAME=solaris9 elif [ "$SYSNAME" = "freebsd" ] ; then SYSNAME=freebsd
else
echo Your system name is $SYSNAME echo We do not know how to build for this kind of system echo Send a note to axiom-***@nongnu.org about it echo exit 0 fi fi } # This function checks for the gawk command. # If it exists then AWKNAME is the complete pathname checkgawk() { AWKNAME=which gawk 2>>trace if [ -n "$AWKNAME" ] ; then
if [ -x $AWKNAME ] ; then echo fi fi } # This function checks for the nawk command. # If it exists then AWKNAME is the complete pathname checknawk() { AWKNAME=which nawk 2>>trace if [ -n "$AWKNAME" ] ; then
if [ -x $AWKNAME ] ; then echo fi fi } # This function checks for the awk command. # If it exists then AWKNAME is the complete pathname checkawk() { AWKNAME=which awk 2>>trace if [ -n "$AWKNAME" ] ; then
if [ -x $AWKNAME ] ; then echo fi fi } # This function uses the check*awk functions to decide # whether the system can build noweb. If one of gawk, nawk or awk # are not found we fail. needAwk () { checkgawk if [ -z "$AWKNAME" ] ; then
checknawk
if [ -z "$AWKNAME" ] ; then checkawk if [ -z "$AWKNAME" ] ; then
echo We need the commands gawk, nawk, or awk
exit 0
fi
fi
fi
}

# The mustSet function tells the user what needs to be typed on the
# command line. If any extra variables need to be set we add them here.
# Currently the only thing we check if for the presence of gawk, which
# is the default in the Makefile. If gawk does not exist we can use
# either nawk or awk but the user has to specify that on the command line.

# We check the system we are using with the uname command and try to
# generate the appropriate value. We fail otherwise.

# We generate the appropriate command line that the user should use.

mustSet() {
echo
echo ===================================================
echo You must set your AXIOM and PATH variables. Type:
echo
echo To build the rest of the system type:
echo
echo export AXIOM=pwd/mnt/$SYSNAME echo 'export PATH=$AXIOM/bin:$PATH' if [ "$SYSNAME" = "freebsd" ] ; then
echo Note that freebsd usually has noweb available
echo If you wish to use the standard version you must type
echo touch noweb
echo If you wish to use a pre-installed GCL you must type
echo make GCLVERSION=gcl-system
fi
if [ "$SYSNAME" = "solaris9" ] ; then echo make AWK=gawk TAR=gtar PATCH=gpatch elif [ "basename$AWKNAME" = "gawk" ] ;
then echo make
else echo make AWK=$AWKNAME fi echo echo configure finished. } ######################################################################### # This is the main line of configure logic. # (1) We test to see if we understand this system name. So far # the recognized strings from uname -s are translated as: # Linux --> linux # MINGW32_NT-5.1 --> windows # SunOS --> Solaris9 # Fedora Core 3 --> fedora3 # freebsd --> freebsd # (1) We test for the AWK variable. We need one of gawk, nawk, or awk # in order to build the noweb software. # (2) Then we output the final message for the user. # # The solaris platform needs patch->gpatch, awk->gawk, tar->gtar ######################################################################### sysname needAwk if [ "x$AXIOM" = "x" ] ;
then mustSet
else
if [ ! "dirname $AXIOM" = "pwd/mnt" ] then mustSet else echo Configure complete. Now type echo echo make echo fi fi \end{verbatim} \section{Basic Setup} \subsection{Autoconf Initialization} The Autoconf machinery needs to be initialized with several pieces of information: \begin{itemize} \item the \emph{name} of the system --- Axiom silver branch'' \item its \emph{version}. I choose to use the date of last checkin. It should probably include the revision number so as to unambiguously identify which Axiom flavour du jour is being built; \item and where to send feedback, \emph{e.g.} bug reports. I have chosen the \email{axiom-developer} list. That could change in the future if we reach a high volume traffic. For the moment, we don't seem to suffer from traffic... \end{itemize} <<Autoconf init>>= AC_INIT([Axiom silver branch], [2006-07-15], [axiom-***@nongnu.org]) @ Autoconf needs some auxilary files that are present in the sub-directory \file{config}. Autoconf needs to be told about that. <<auxiliary config files>>= AC_CONFIG_AUX_DIR(config) @ Notice that since we don't use Automake (yet), we don't initialize the Automake subsystem. <<Automake init>>= # AM_INIT_AUTOMAKE([foreign]) @ We require Autoconf 2.59 or higher from the developer part. Please, note that this is no requirement on the user build environment. All, it means is that if someone makes changes to the current \file{configure.ac} file, that someone needs to have Autoconf 2.59 or higher to process this file in order to regenerate \file{configure}. <<Autoconf prerequirement>>= AC_PREREQ([2.59]) @ \subsection{Source tree sanity check} The Autoconf system implements a very basic, simple-minded, sanity check whereby it will refuse to run \file{configure} if the source tree does not contain a specified file, that serves a witness for a bona fide source tree. Here, I have chosen \file{Makefile.pamphlet} from the \file{src} subdirectory. <<sanity check>>= AC_CONFIG_SRCDIR(src/Makefile.pamphlet) @ \subsection{Build Environment} Standard build environments consist of \begin{enumerate} \item the \emph{host} platform, \item the \emph{build} platform, and \item the \emph{target} platform. \end{enumerate} FIXME: Example on these notions. We need to get the canonical names for the above three platforms. After call to this macro, those values are available in the variables [[host]], [[build]], and [[target]], respectively. <<host build target platfoms>>= AC_CANONICAL_SYSTEM @ For the moment, Axiom supports neither cross-compilation, nor Canadian cross. Consequently, we must bail out if the build is not native. <<check cross-compilation>>= if test$host != $build -o$host != $target; then AC_MSG_ERROR([Sorry, only native builds are currently supported]) fi @ Most of the tools we're testing for are with respect to the build environment, neither the host nor the target. First of all, check for a C compiler --- \textsl{GCC/gcc} preferably. As written, this test is OK because currently we support only native builds. However, it needs to be more carefully written when we move to cross-compilation. <<check for C compiler>>= AC_PROG_CC @ Then, check for a usable 'install' program. <<check for install>>= AC_PROG_INSTALL @ The old build machinery needs 'awk'. Currently, it checks for 'gawk', 'nawk', and 'awk'. Autoconf has a predefined test for that task. It checks for 'gawk', 'mawk', 'nawk', and 'awk' in that order. That should be OK and match Axiom's need. The old build system claims that on solaris9, gawk, gtar and gpatch are required (with no much explanation of why). Notice that these programs are needed only to build Axiom; so we do check based on the value of [[build]]. <<check for awk tar patch>>= case${build} in
*-solaris9)
AC_CHECK_PROG([AWK], [gawk],
[gawk], [AC_MSG_ERROR([Axiom needs gawk])])

AC_CHECK_PROG([TAR], [gtar],
[gtar], [AC_MSG_ERROR([Axiom needs gtar])])

AC_CHECK_PROG([PATCH], [gpatch],
[gptach], [AC_MSG_ERROR([Axiom needs gpatch])])
;;

*)
AC_PROG_AWK

AC_CHECK_PROGS([TAR], [gtar tar],
[AC_MSG_ERROR([Axiom needs a tar program])])

AC_CHECK_PROGS([PATCH], [gpatch patch],
[AC_MSG_ERROR([Axiom needs a patch program])])
;;
esac
@

Once we have found those values, we must substitute them in the
Makefiles. For the moment, these actions are effectless as the
current makefile are hand-written and Autoconf-unaware.
<<substitute AWK TAR PATCH>>=
AC_SUBST(AWK)
AC_SUBST(TAR)
AC_SUBST(PATCH)
@

Obviously we need the 'make' program. No build will proceed without
it.
<<check for make>>=
AC_CHECK_PROG([MAKE], [make],
[make], [AC_MSG_ERROR([make' program missing.])])
@

Here, we replicate the behaviour of the old configure, waiting for
a better alternative, e.g. where it is not needed. THIS WILL BE
REMOVED IN THE NEAR FUTURE.

First, the old machinery has a very coarse target name "discovery"
<<replicate old behaviour>>=
if test -f /etc/redhat-release; then
SYSNAME=cat /etc/redhat-release
if test "$SYSNAME" = "Fedora Core release 3 (Heidelberg)"; then SYSNAME=fedora3 fi AC_MSG_NOTICE([SYSNAME=$SYSNAME])
fi
if test "$SYSNAME" != "fedora3"; then SYSNAME=uname -s AC_MSG_NOTICE([$SYSNAME])
case "$SYSNAME" in Linux) SYSNAME=linux ;; MINGW32_NT-5.1) SYSNAME=windows ;; SunOS) SYSNAME=solaris9 ;; freebsd) ;; *) AC_MSG_NOTICE([Your system name is$SYSNAME])
AC_MSG_NOTICE([We do not know how to build for this kind of system])
AC_MSG_ERROR([Send a note to axiom-***@nongnu.org about it])
esac
fi

must_set_AXIOM() {
AC_MSG_NOTICE([])
AC_MSG_NOTICE([===================================================])
AC_MSG_NOTICE([])
AC_MSG_NOTICE([You must set your AXIOM and PATH variables. Type:])
AC_MSG_NOTICE([])
AC_MSG_NOTICE([export AXIOM=pwd/mnt/$SYSNAME]) AC_MSG_NOTICE([export PATH=\$AXIOM/bin:\$PATH]) case "$SYSNAME" in
freebsd)
AC_MSG_NOTICE([Note that freebsd usually has noweb available])
AC_MSG_NOTICE([echo If you wish to use the standard version you must type])
AC_MSG_NOTICE([touch noweb])
AC_MSG_NOTICE([If you wish to use a pre-installed GCL you must type])
AC_MSG_NOTICE([make GCLVERSION=gcl-system])
;;
solaris9)
AC_MSG_NOTICE([make AWK=gawk TAR=gtar PATCH=gpatch])
;;
*)
AC_MSG_NOTICE([make AWK=$AWK]) esac } if test "x$AXIOM" = "x"; then
must_set_AXIOM
elif test "dirname $AXIOM" != "pwd/mnt"; then must_set_AXIOM else AC_MSG_NOTICE([configure complete. Now type ]) AC_MSG_NOTICE([ ]) AC_MSG_NOTICE([make]) AC_MSG_NOTICE([ ]) AC_MSG_NOTICE([export AXIOM=pwd/mnt/$SYSNAME])
AC_MSG_NOTICE([export PATH=\$AXIOM/bin:\$PATH])
fi
@

<<*>>=
<<Autoconf init>>

<<auxiliary config files>>

<<Automake init>>

<<Autoconf prerequirement>>

<<sanity check>>

<<host build target platfoms>>

<<check cross-compilation>>

<<check for C compiler>>

<<check for install>>

<<check for awk tar patch>>

<<substitute AWK TAR PATCH>>

<<check for make>>

<<replicate old behaviour>>
@

\end{document}
Ralf Hemmecke
2006-08-02 10:26:55 UTC
Hi Gaby,

your working times are remarkable. Aren't you in Texas? ;-)

I have seen that there are now also ChangeLog files under axiom/src/doc
and axiom/src/boot. Is it common to have *not* just one top-level
ChangeLog? Now I am getting confused of where I should enter my
changelog stuff.

Ralf
Gabriel Dos Reis
2006-08-02 10:45:13 UTC
Ralf Hemmecke <***@hemmecke.de> writes:

| Hi Gaby,
|
| your working times are remarkable. Aren't you in Texas? ;-)

Yes. One nice thing about having a baby who is growing a tooth is
that you for sure will be awaken some nights :-)

| I have seen that there are now also ChangeLog files under
| axiom/src/doc and axiom/src/boot. Is it common to have *not* just one
| top-level ChangeLog? Now I am getting confused of where I should enter
| my changelog stuff.

I believe both ways of doing things are common; I have a slight
preference for a ChangeLog file in each directory -- having practiced
both ways. That way I don't have long file names eating up all space
just to give me little clues.

For the Silver branch, you will find a ChangeLog file in each
directory (almost). Juts put your descriptions there.

For the branches, it is actually more practical to have a different
ChangeLog files -- the convention I use is ChangeLog.<branch-name>.
There are several paratical reasons for that:
(1) avoid spurious conflicts when merging;
(2) a glance at the file gives an overview of changes made on
the branch.

Ah; I've just finished the "./configure && make && make install" stuff.
I'll commit after testing.

-- Gaby
Ralf Hemmecke
2006-08-02 10:44:09 UTC
Post by Gabriel Dos Reis
Based on the feedback I got, I implemented the Autoconf-based
configuration stuff as a pamphlet file (see below).
I think, Bill's suggestion is the right one. We have a pure ./configure
file that is distributed with the Axiom sources and a configure.ac.pamphlet.

We have two ways to compile Axiom:

1) For ordinary users who want to download and compile and run axiom
Post by Gabriel Dos Reis
./configure
make
make install
axiom

should be enough.

2) For developers who want to compile Axiom change it and test their
stuff. (Same commands as above.)

3) For developers who want to work on the build process and need to
change configure.ac.pamphlet. Here

./configure
make configure
autoconf configure.ac > configure
./configure
make
make install

should do the compilation. The first configure (that comes with Axiom)
checks for noweb and generates a simple Makefile that enables a
"make configure". "make configure" could then call notangle to generate
configure.ac (It could even include "autoconf configure.ac >
configure".) Would that be a bad bootstrapping?

Gaby, it is only you who must run

notangle configure.ac.pamphlet > configure.ac
autoconf configure.ac > configure

ONCE and then we see a "configure" file in the distribution of Axiom.

Ralf
Gabriel Dos Reis
2006-08-02 11:17:27 UTC
Ralf Hemmecke <***@hemmecke.de> writes:

| > Based on the feedback I got, I implemented the Autoconf-based
| > configuration stuff as a pamphlet file (see below).
|
| I think, Bill's suggestion is the right one. We have a pure ./configure
| file that is distributed with the Axiom sources and a configure.ac.pamphlet.

from the outset, the project aimed at shipping ./configure. However,
that file must be generated from somewhere. For Autoconf, that must
be something like configure.ac.
The debate was whether configure.ac is sufficient by itself as
literate or whether it must come from a pamphlet file.
My opinion is that it is sufficient. Others disagree. So, I
implemented the solution others suggested -- even though I disagree
with it.

| We have two ways to compile Axiom:
|
| 1) For ordinary users who want to download and compile and run axiom
|
| > ./configure
| > make
| > make install
| axiom
|
| should be enough.
|
| 2) For developers who want to compile Axiom change it and test their
| stuff. (Same commands as above.)
|
| 3) For developers who want to work on the build process and need to
| change configure.ac.pamphlet. Here
|
| ./configure
| make configure

That is too circular and too complicated. It is a good recipe for subtle
bugs. We don't want to be too clever at that level.

| autoconf configure.ac > configure
| ./configure
| make
| make install

Currently, here is the sequence I propose -- and I'm using right now

(1) modify configure.ac.pamphlet
(2) run ./build-setup.sh -- you don't need to know what
-- it does, except it does the
-- right thing for you
(3) ./configure && make && make install

|
| should do the compilation. The first configure (that comes with Axiom)
| checks for noweb and generates a simple Makefile that enables a
| "make configure". "make configure" could then call notangle to
| generate configure.ac (It could even include "autoconf configure.ac >
| configure".) Would that be a bad bootstrapping?
|
| Gaby, it is only you who must run
|
| notangle configure.ac.pamphlet > configure.ac
| autoconf configure.ac > configure

I do expect other developers making changes to
configure.ac.pamphlet (and later to any Makefile.am.pamphlet) to run
equivalent commands. Fortunately, that is all taken care of by
build-setup.sh. :-)

build-setup.sh is not an end-user stuff. It is only for maintainers
of the build machinery.

End users just follow the scenario at 1)

Mainitainers are required to have noweb installed -- so I proceeded to
install noweb myself.

Bill suggested we don't keep configure.ac. I fear we must kepp it --
for the purpose of preventing the build machinery maintainers from
introducing subtle bugs. Components of Autotools have dependency on
such files in several circumstances.

-- Gaby
Ralf Hemmecke
2006-08-02 13:03:30 UTC
Hi Gaby,
Post by Gabriel Dos Reis
| 3) For developers who want to work on the build process and need to
| change configure.ac.pamphlet. Here
|
| ./configure
| make configure
That is too circular and too complicated. It is a good recipe for subtle
bugs. We don't want to be too clever at that level.
| autoconf configure.ac > configure
| ./configure
| make
| make install
Currently, here is the sequence I propose -- and I'm using right now
(1) modify configure.ac.pamphlet
(2) run ./build-setup.sh -- you don't need to know what
-- it does, except it does the
-- right thing for you
(3) ./configure && make && make install
First of all let me say that I have no idea how all this
autoconf/automake works... So I expect to learn something through this

I don't see a difference between
Post by Gabriel Dos Reis
(1) modify configure.ac.pamphlet
(2) run ./build-setup.sh
and my suggested

modify configure.ac.pamphlet
make configure

In my previous mail I suggested "./configure && make configure" but that
was only to make sure that "noweb" is found by "configure" if you have
it installed somewhere, then of course you don't need to run 'configure'.

Now whether you call "build-setup.sh" or "make configure" isn't much of
a difference. Is it? It is responsible for creating configure.ac from
configure.ac.pamphlet.

I know that "make" checks whether the "Makefile" is as new as it could
be. But does it check whether there is Makefile.in or Makefile.am and
re-generates the Makefile from that???

And why do you insist on having all Makefiles generated during
"configure"? For me it is perfectly fine if the Makefiles are
regenerated from their .pamphlets during the build of Axiom. It is only
that the Makefiles should more rely on variables that define programs
like using ${TAR} instead of just tar. Or is the whole thing more complicated than I think? Post by Gabriel Dos Reis | Gaby, it is only you who must run | | notangle configure.ac.pamphlet > configure.ac | autoconf configure.ac > configure I do expect other developers making changes to configure.ac.pamphlet (and later to any Makefile.am.pamphlet) to run equivalent commands. Fortunately, that is all taken care of by build-setup.sh. :-) So each time I modify a file which is called configure.ac.pamphlet or Makefile.am.pamphlet, I should run "build-setup.sh". What if I forget that? There should be at least a warning telling me that I should run "build-setup.sh". Post by Gabriel Dos Reis build-setup.sh is not an end-user stuff. It is only for maintainers of the build machinery. That's clear. Post by Gabriel Dos Reis Bill suggested we don't keep configure.ac. I fear we must kepp it -- for the purpose of preventing the build machinery maintainers from introducing subtle bugs. Components of Autotools have dependency on such files in several circumstances. Can you make up an example so that we see what problems can occur and appreciate why we should keep configure.ac. Then that should be documented (in a pamphlet file) describing the build process. Currently, I also don't feel need for configure.ac, given that maintainers have noweb and configure.ac.pamphlet. Ralf Gabriel Dos Reis 2006-08-02 14:58:44 UTC Permalink Ralf Hemmecke <***@hemmecke.de> writes: [...] | I don't see a difference between | | > (1) modify configure.ac.pamphlet | > (2) run ./build-setup.sh | | and my suggested | | modify configure.ac.pamphlet | make configure "make" means that there is a Makefile that is used to make "configure". There isn't. Any makefile must be generated at configure time when building Axiom. I know that is not what the current system does; but it is a bug -- it got me several times. | In my previous mail I suggested "./configure && make configure" but | that was only to make sure that "noweb" is found by "configure" if you | have it installed somewhere, then of course you don't need to run | 'configure'. | | Now whether you call "build-setup.sh" or "make configure" isn't much | of a difference. Is it? from my perspective; there is. See above. [...] | I know that "make" checks whether the "Makefile" is as new as it could | be. But does it check whether there is Makefile.in or Makefile.am and | re-generates the Makefile from that??? No. Makefile.in usually contains variables that need to be substituted. Makefile.am is even at higher level. You need Automake. Now, when you have the full Autotools in place, if you modify Makefile.am or configure.ac and you say "make", there usually is a dectection and autoconf is run to regenerate | And why do you insist on having all Makefiles generated during | "configure"? Because they contain platform-specific information and the standard way to reduce duplication and make the whole system manageable is to abstract over the variabilities. Otherwise, we have the current mess. This isn't an innovation. Look at any other system that uses Autoconf + Automake (Maxima is one that additionally has a Lisp-based configure). | For me it is perfectly fine if the Makefiles are | regenerated from their .pamphlets during the build of Axiom. It is | only that the Makefiles should more rely on variables that define | programs like using${TAR} instead of just tar.
|
| Or is the whole thing more complicated than I think?

In part yes.

The only benefit I see of not storing configure.ac is more
complications and more potential for bugs. I don't see the benefits.

| > | Gaby, it is only you who must run
| > | | notangle configure.ac.pamphlet > configure.ac
| > | autoconf configure.ac > configure
| > I do expect other developers making changes to
| > configure.ac.pamphlet (and later to any Makefile.am.pamphlet) to run
| > equivalent commands. Fortunately, that is all taken care of by
| > build-setup.sh. :-)
|
| So each time I modify a file which is called configure.ac.pamphlet or
| Makefile.am.pamphlet, I should run "build-setup.sh".

Yes.

| What if I forget that?

I'll catch you. Your "make configure" suffers from the same problem.

When we move to Automake, the system will assist me.

| There should be at least a warning telling me that I should run
| "build-setup.sh".

Probably; that is low-priority right now.

|
| > build-setup.sh is not an end-user stuff. It is only for maintainers
| > of the build machinery.
|
| That's clear.
|
| > Bill suggested we don't keep configure.ac. I fear we must kepp it --
| > for the purpose of preventing the build machinery maintainers from
| > introducing subtle bugs. Components of Autotools have dependency on
| > such files in several circumstances.
|
| Can you make up an example so that we see what problems can occur and
| appreciate why we should keep configure.ac.

No, I don't have time for that -- I'm sorry. You have to take my word
for it for the moment. When I get more time, I can play with testcases.

| Then that should be
| documented (in a pamphlet file) describing the build process.

Yes. That is not an issue.

| Currently, I also don't feel need for configure.ac, given that
| maintainers have noweb and configure.ac.pamphlet.

Then just forget about it and run build-setup.sh.

-- Gaby
Ralf Hemmecke
2006-08-02 21:09:47 UTC
Hi Gaby,

I'm sorry if I steal your precious time by my stupid questions...
Post by Gabriel Dos Reis
[...]
| I don't see a difference between
|
| > (1) modify configure.ac.pamphlet
| > (2) run ./build-setup.sh
|
| and my suggested
|
| modify configure.ac.pamphlet
| make configure
"make" means that there is a Makefile that is used to make "configure".
There isn't. Any makefile must be generated at configure time when
building Axiom. I know that is not what the current system does;
but it is a bug -- it got me several times.
That is your point of view and years of experience. I will trust you in
setting up Autotools support for Axiom, since for now that is more
important then sticking to pamphlet style. That you are going to
document everything in some form is clear to me.

But still. I assumed that we distribute the sources with configure and
an initial Makefile. That would be generated first by you via

notangle configure.ac.pamphlet > configure.ac
autoconf configure.ac > configure
./configure
svn commit configure
svn commit Makefile

That Makefile would contain code to generate "configure". So it

If I am an ordinary user, I start with ./configure && make.
If I am more advanced and change some stuff in configure.ac.pamphlet,
I start with "./configure" to make sure that I have "noweb" and then
re-iterate via "make configure && ./configure". I really cannot see a
problem with that.
Post by Gabriel Dos Reis
| I know that "make" checks whether the "Makefile" is as new as it could
| be. But does it check whether there is Makefile.in or Makefile.am and
| re-generates the Makefile from that???
No. Makefile.in usually contains variables that need to be
substituted. Makefile.am is even at higher level. You need
Automake. Now, when you have the full Autotools in place, if you
modify Makefile.am or configure.ac and you say "make", there usually
is a dectection and autoconf is run to regenerate
But I never modify Makefile.am or configure.ac. Only the corresponding
pamphlet files. So there clearly there is a target like

%: %.pamphlet
notangle $< >$@

or something the like.

Ralf
Gabriel Dos Reis
2006-08-02 23:39:33 UTC
Ralf Hemmecke <***@hemmecke.de> writes:

[...]

| > | I know that "make" checks whether the "Makefile" is as new as it could
| > | be. But does it check whether there is Makefile.in or Makefile.am and
| > | re-generates the Makefile from that???
| > No. Makefile.in usually contains variables that need to be
| > substituted. Makefile.am is even at higher level. You need
| > Automake. Now, when you have the full Autotools in place, if you
| > modify Makefile.am or configure.ac and you say "make", there usually
| > is a dectection and autoconf is run to regenerate
|
| But I never modify Makefile.am or configure.ac.

That is right. But Autotools have implicit dependency on those.

-- Gaby
Gabriel Dos Reis
2006-08-13 15:14:51 UTC
Ralf Hemmecke <***@hemmecke.de> writes:

[...]

| So each time I modify a file which is called configure.ac.pamphlet or
| Makefile.am.pamphlet, I should run "build-setup.sh". What if I forget
| that? There should be at least a warning telling me that I should run
| "build-setup.sh".

Ralf --

As of revision 96, if you're in the middle of development, you
modify configure.ac.pamphlet or Makefile.pamphlet, the top level Makefile
will regenerate configure, Makefile.in, and Makefile as appropriate.

-- Gaby

Page, Bill
2006-08-01 21:49:13 UTC
Gaby,
Post by Gabriel Dos Reis
Instead of fiddling at infinitum with the new build machinery on
my local disk. I decided to trim it down to something "simple" to
request feedback about some aspects of the work.
The core goal is to move Axiom's build system to GNU Autotools
which have become quite standard now, in the Unix-world.
Yes! ++1
Post by Gabriel Dos Reis
For that I've converted the current configure file to
configure.ac (which Autoconf will use to regenerate configure).
The main motivation here is factor out many platforms variability
issues and dump them on Autoconf -- who already knows how to solve
them.
Axiom has the requirement that the source files must be
literate -- use of pamphlet files. Autoconf expects its input to
be in particular form. Furthermore, the pamphlet file requirement
introduces an unpleasant bootstrapping issue that will hopefully
be resolved.
Are you referring to the problem of the noweb dependency? I think
this is not so much of a problem if we treat it properly as a
dependency.

configure.ac itself should be a named chunk - probably in the main
Makefile.pamphlet. This might make regenerating ./configure a little
more complex than usual but not much.

Most open source installations ship a pre-generated ./configure
file and I think Axiom should be no different.

The first thing ./configure should do is check for noweb and tell
to ./configure can be to build and install noweb from the sources
supplied with Axiom. But in general Axiom should not be in the
Post by Gabriel Dos Reis
So, in its current form, I put the literate stuff in comments, so
that if you do
cat configure.ac | sed -e 's/^## //' > configure.tex
you can an almost LaTeX file. "Almost" because LaTeX commands need
to start with a backslash. That character used to be reserved for
line continuation in shell world. So to avoid any surprise, I used
the percent character (which unfortunately is LaTeX comment
character). So to get the LaTeX document, one needs to apply another
sed that replaces the percent character with the backslash character.
I've tested that I can feed the file to Autoconf, get a configure
file out of it and launch a successful build.
This work is being done on the "build-improvements" branch -- and it
has not been committed there yet.
Before painting myself into a corner, I would like to have feedbacks
from all y'all.
All this looks like a horrible hack to me - unless I am missing
something fairly basic. What is wrong with the usual way of using
noweb?

Regards,
Bill Page.
Gabriel Dos Reis
2006-08-01 22:49:23 UTC
"Page, Bill" <***@drdc-rddc.gc.ca> writes:

[...]

| > For that I've converted the current configure file to
| > configure.ac (which Autoconf will use to regenerate configure).
| > The main motivation here is factor out many platforms variability
| > issues and dump them on Autoconf -- who already knows how to solve
| > them.
| >
| > Axiom has the requirement that the source files must be
| > literate -- use of pamphlet files. Autoconf expects its input to
| > be in particular form. Furthermore, the pamphlet file requirement
| > introduces an unpleasant bootstrapping issue that will hopefully
| > be resolved.
|
| Are you referring to the problem of the noweb dependency? I think
| this is not so much of a problem if we treat it properly as a
| dependency.

Currently, it if one modifies the toplevel Makefile.pamphlet, one also
needs to regenerate the corresponding Makefile. I don't want that.

All Makefiles should be generated at configured time.

Furthermore, one needs to have noweb in order to regenerate it. If
the target system needs to build noweb, we are hosed. We don't want.

All that is elimintated by not having to regenarate Makefile before
actual configuration.

| configure.ac itself should be a named chunk - probably in the main
| Makefile.pamphlet. This might make regenerating ./configure a little
| more complex than usual but not much.

configure will swamp the generation of all Makefiles from the
corresponding pamphlets. Consequently I don't want to tie the
generation of configure to Makefiles that way. We don't need that
bootstrapping problem.

| Most open source installations ship a pre-generated ./configure
| file and I think Axiom should be no different.

Yes, that is what we will do. That configure is generated by Autoconf
based on configure.ac. Autoconf does not expect pamphlet files.
Autoconf want an input in a language it understands. Consequently, we
must present configure.ac in a way appropriate for Autoconf.
Therefore we cannot depend on noweb (notangle).

In my current system, the only dependency of configure on makefiles is
when the makefiles are "touched" in a way that needs reconfiguration;
and way to extract the documentation out of configure.ac.

| The first thing ./configure should do is check for noweb and tell

Yes, that is what is done in my current local tree. That is one
reason why we don' want configure.ac to depends on noweb.

| One "last resort" option
| to ./configure can be to build and install noweb from the sources
| supplied with Axiom. But in general Axiom should not be in the

That is also an implemented option in my current system.

| > So, in its current form, I put the literate stuff in comments, so
| > that if you do
| >
| > cat configure.ac | sed -e 's/^## //' > configure.tex
| >
| > you can an almost LaTeX file. "Almost" because LaTeX commands need
| > to start with a backslash. That character used to be reserved for
| > line continuation in shell world. So to avoid any surprise, I used
| > the percent character (which unfortunately is LaTeX comment
| > character). So to get the LaTeX document, one needs to apply another
| > sed that replaces the percent character with the backslash character.
| >
| > I've tested that I can feed the file to Autoconf, get a configure
| > file out of it and launch a successful build.
| >
| > This work is being done on the "build-improvements" branch -- and it
| > has not been committed there yet.
| >
| > Before painting myself into a corner, I would like to have feedbacks
| > from all y'all.
| >
|
| All this looks like a horrible hack to me - unless I am missing
| something fairly basic. What is wrong with the usual way of using
| noweb?

See the explanations above.

This is a hack; I agree. No more horrible than what we currently do
with noweb. However, I don't like it. That is why I called for
discussion and suggestions for improvements :-)

To sum up.

We need

(1) to write configure.ac directly in a language understood by Autoconf;
(2) a way to get the documentation out of configure.ac
(3) no dependency of configure.ac (and therefore configure) on noweb.

For the long term, all Makefiles should be generated at configure time.

-- Gaby
Cliff Yapp
2006-08-02 02:11:51 UTC
Post by Gabriel Dos Reis
| Most open source installations ship a pre-generated ./configure
| file and I think Axiom should be no different.
Yes, that is what we will do. That configure is generated by Autoconf
based on configure.ac. Autoconf does not expect pamphlet files.
Autoconf want an input in a language it understands. Consequently, we
must present configure.ac in a way appropriate for Autoconf.
Therefore we cannot depend on noweb (notangle).
If I understand the question, you are asking how we can have only literate
program files in Axiom at the get go and still not depend on noweb. The
Post by Gabriel Dos Reis
In my current system, the only dependency of configure on makefiles is
when the makefiles are "touched" in a way that needs reconfiguration;
and way to extract the documentation out of configure.ac.
I suppose I'm dense, but I thought it would end up like this:

1) Have configure.ac and configure.ac.pamphlet both present in the top level
directory. configure.ac.pamphlet is where editing is done, and when a commit
is made (for this one file) a configure.ac file is also committed (or even
autogenerated by the server, if we want to go so far as to configure it to do
so.)

2) Extracting the documentation out of configure.ac.pamphlet would require
the use of the notangle command, or once we have a lisp noweb using that
tool.

I've always wondered why LaTeX itself couldn't handle processing a pamphlet
file as a regular latex file - after all, couldn't it just replace
<<chunckname>> [content] @ by a fancy version of \begin{verbatium}[content]
\end{verbatium} or some such? You would still need a notangle command to
extract source code, but at least you could read the documentation without
needing anything more than LaTeX.

I suppose I'm missing features noweb gives us though.
Post by Gabriel Dos Reis
| The first thing ./configure should do is check for noweb and tell
Yes, that is what is done in my current local tree. That is one
reason why we don' want configure.ac to depends on noweb.
That's why we will need to have both configure.ac and configure.ac.pamphlet.
This will always be true if we want to avoid having the end user manually use
some other tool to extract the first file, whether it is configure.ac or some
other file. As long as configure.ac.pamphlet is written and maintained in
the .pamphlet file I don't see a problem with special casing the continual
presence of configure.ac - it's pretty much inevitable.
Post by Gabriel Dos Reis
| All this looks like a horrible hack to me - unless I am missing
| something fairly basic. What is wrong with the usual way of using
| noweb?
See the explanations above.
I understand the desire to be able to bootstrap, but there is no way around
it - we must have one non-pamphlet file present to kickstart the process.
Personally I would suggest two or three - a set of Bash and Windows script
files containing all the autoconf and other incantantations the user would
need to make might simplify matters even more.

I don't think we should fear a couple of non-pamphlet files being present in
the top level directory - indeed, it will simplify matters considerably from
the user standpoint. They won't know (and may not want to know) what a
pamphlet file is all about - they just want to build and use the math program
to do math. In that case, the complete absence of anything familiar will
turn them off.

I would suggest the minimal (and maybe maximal) set of:

configure.ac
buildaxiom.sh
buildaxiom_windows.<whatever windows uses>

README should be in basic, no frills ASCII text - about as universally
readable as you can get. It should explain a little about Axiom, and how to
get started building it (tools needed, etc.)
Post by Gabriel Dos Reis
This is a hack; I agree. No more horrible than what we currently do
with noweb. However, I don't like it. That is why I called for
discussion and suggestions for improvements :-)
There is no way to bootstrap without a "lower level" file present to kick the
process off, unless (as you already attempted) the higher level files are
also legal lower level files. I don't think we want to redefine the literate
strategy we use to that degree - it's not worth it.

In a way, it's the same problem Tim faced with Axiom being needed to build
Axiom. The only way out was (and still is) to have some files in lisp
present, to teach lisp to read and understand BOOT and SPAD. Same deal here.
It's a universal problem - if we wanted to bootstrap a machine from a cold,
memoryless, powerless state we would need some binary to feed it before it
could start speaking assembly. True bootstrapping is a very rare event ;-).
Normally even new platforms have their software cross compiled from other
platforms, since the human beings involved don't want to make the translation
themselves. Anyway, that's another topic.
Post by Gabriel Dos Reis
To sum up.
We need
(1) to write configure.ac directly in a language understood by Autoconf;
(2) a way to get the documentation out of configure.ac
(3) no dependency of configure.ac (and therefore configure) on noweb.
1 and 3 are solved by always having a current notangle of configure.ac present
in the tree at all times. As for two, I think we should assume a working
notangle is available. We have to assume SOMETHING is working, and since the
generated document needs LaTeX to be processed I think the additional
assumption of notangle (particularly if configure.ac can build it) is
acceptable.
Post by Gabriel Dos Reis
For the long term, all Makefiles should be generated at configure time.
Agreed.

Cheers,
CY
Gabriel Dos Reis
2006-08-02 03:02:01 UTC
Cliff Yapp <***@yahoo.com> writes:

| On Tuesday 01 August 2006 6:49 pm, Gabriel Dos Reis wrote:
|
| > | Most open source installations ship a pre-generated ./configure
| > | file and I think Axiom should be no different.
| >
| > Yes, that is what we will do. That configure is generated by Autoconf
| > based on configure.ac. Autoconf does not expect pamphlet files.
| > Autoconf want an input in a language it understands. Consequently, we
| > must present configure.ac in a way appropriate for Autoconf.
| > Therefore we cannot depend on noweb (notangle).
|
| If I understand the question, you are asking how we can have only literate
| program files in Axiom at the get go and still not depend on noweb. The
| simple answer is, we can't.

Actually, we can if we don't insist that literate files must be
pamphlet files or we must use noweb :-)

[...]

| I've always wondered why LaTeX itself couldn't handle processing a pamphlet
| file as a regular latex file - after all, couldn't it just replace
| <<chunckname>> [content] @ by a fancy version of \begin{verbatium}[content]
| \end{verbatium} or some such? You would still need a notangle command to
| extract source code, but at least you could read the documentation without
| needing anything more than LaTeX.

I would like to move away from the @ and [] fancy syntax. But, that
is a different battle I guess.

[...]

| As long as configure.ac.pamphlet is written and maintained in
| the .pamphlet file I don't see a problem with special casing the continual
| presence of configure.ac - it's pretty much inevitable.

OK. I'll write up a development enrivonment minimal requirements and
include noweb (I still dislike that specific requirement).

| > | All this looks like a horrible hack to me - unless I am missing
| > | something fairly basic. What is wrong with the usual way of using
| > | noweb?
| >
| > See the explanations above.
|
| I understand the desire to be able to bootstrap, but there is no way around
| it - we must have one non-pamphlet file present to kickstart the process.
| Personally I would suggest two or three - a set of Bash and Windows script
| files containing all the autoconf and other incantantations the user would
| need to make might simplify matters even more.
|
| I don't think we should fear a couple of non-pamphlet files being present in
| the top level directory - indeed, it will simplify matters considerably from
| the user standpoint. They won't know (and may not want to know) what a
| pamphlet file is all about - they just want to build and use the math program
| to do math. In that case, the complete absence of anything familiar will
| turn them off.
|
| I would suggest the minimal (and maybe maximal) set of:
|
| configure.ac
| buildaxiom.sh
| buildaxiom_windows.<whatever windows uses>

The minimal setting should include

configure -- generated by Autoconf; end-user stuff
configure.ac -- generated by noweb, developer
configure.ac.pamphlet -- hand-edited, developer
build-setup.sh -- hand-edited; end-user stuff

build-setup.sh will abstract over all the low-level commands that
generate configure.ac from configure.ac.pamphlet, configure from
configure.ac -- and Makefile.am from Makefile.am.pamphlet when we
converts to Automake.

I'm not a windows developer, so I'll leave the windows stuff to others
(hi Bill).

[...]

| > This is a hack; I agree. No more horrible than what we currently do
| > with noweb. However, I don't like it. That is why I called for
| > discussion and suggestions for improvements :-)
|
| There is no way to bootstrap without a "lower level" file present to kick the
| process off,

I know. configure.ac is that file. And the point that started the
experiment is that once you've written configure.ac, there is very
little gained by putting it in a pamphlet file. All the documentation
will need to be present in the final generated configure file.
Consequently, the low-level stuff here equals to the high-level variant.

| unless (as you already attempted) the higher level files are
| also legal lower level files. I don't think we want to redefine the literate
| strategy we use to that degree - it's not worth it.

I'm really annoyed by the dependency on noweb to bootstrap the
process. I would rather, by a large degree, vanilla traditional
tools, e.g. plain 'sed'. However, I will proceed as outlined above, and
will at the same time respectfully register my strong disagreement.

[...]

| In a way, it's the same problem Tim faced with Axiom being needed to build
| Axiom. The only way out was (and still is) to have some files in lisp
| present, to teach lisp to read and understand BOOT and SPAD. Same deal here.
That is a good recipe for maintenance nightmarre -- recent debates about
redundant boot anyone? There ought to be a better way to deal with it.

In the case of configure.ac, I'm pretty convinced because I've tried
several ways. The conclusion invariably is that there is no
difference between the low-level stuff and the high-level stuff. Any
attempt to raise the abstraction there is a non-interesting, fruitless
exercise.

The case for boot might be different though -- I've not spent enough
time there yet.

| It's a universal problem - if we wanted to bootstrap a machine from a cold,
| memoryless, powerless state we would need some binary to feed it before it
| could start speaking assembly. True bootstrapping is a very rare event ;-).

I've been hacking real life compilers for over decade now :-)

[...]

| 1 and 3 are solved by always having a current notangle of configure.ac present
| in the tree at all times. As for two, I think we should assume a working
| notangle is available. We have to assume SOMETHING is working,

No question we have to assume something is workin -- just assume,
everybody knows that.

The issue is whether it must be noweb. I think "no". But, I do

-- Gaby
Cliff Yapp
2006-08-02 12:06:51 UTC
Post by Gabriel Dos Reis
Actually, we can if we don't insist that literate files must be
pamphlet files or we must use noweb :-)
OK, point...
Post by Gabriel Dos Reis
| As long as configure.ac.pamphlet is written and maintained in
| the .pamphlet file I don't see a problem with special casing the
| continual presence of configure.ac - it's pretty much inevitable.
OK. I'll write up a development enrivonment minimal requirements and
include noweb (I still dislike that specific requirement).
We have discussed alternatives to noweb before, including writing our own Lisp
version of noweb. I expect eventually we will be able to move off of noweb,
but right now it's easier to use what works.
Post by Gabriel Dos Reis
| unless (as you already attempted) the higher level files are
| also legal lower level files. I don't think we want to redefine the
| literate strategy we use to that degree - it's not worth it.
I'm really annoyed by the dependency on noweb to bootstrap the
process. I would rather, by a large degree, vanilla traditional
tools, e.g. plain 'sed'. However, I will proceed as outlined above, and
will at the same time respectfully register my strong disagreement.
Oh, no question I agree that depending specifically on noweb is unfortunate.
Post by Gabriel Dos Reis
| In a way, it's the same problem Tim faced with Axiom being needed to
| build Axiom. The only way out was (and still is) to have some files in
| lisp present, to teach lisp to read and understand BOOT and SPAD. Same
| deal here.
That is a good recipe for maintenance nightmarre -- recent debates about
redundant boot anyone? There ought to be a better way to deal with it.
I think the way out is clear - put all of the low level logic in Lisp, and the
high level logic in Aldor (or SPAD, until Aldor is available). Tim has begun
that process with bookvol5 - once all of interp is in human readable lisp,
both the boot files in interp and the boot directory itself should become
unnecessary. It should also enable a lot of other interesting activities
with existing Lisp tools, but that's a side benefit.
Post by Gabriel Dos Reis
| It's a universal problem - if we wanted to bootstrap a machine from a
| cold, memoryless, powerless state we would need some binary to feed it
| before it could start speaking assembly. True bootstrapping is a very
| rare event ;-).
I've been hacking real life compilers for over decade now :-)
Sorry, didn't mean to lecture the experts ;-). I just find it a really
interesting problem - the answer to "how do we go from cold, dumb, inert
lumps of metal and silicon to a running computer" is a lot less trivial than
one might think - the technological Dr. Frankenstein, so to speak ;-)
Post by Gabriel Dos Reis
| 1 and 3 are solved by always having a current notangle of configure.ac
| present in the tree at all times. As for two, I think we should assume a
| working notangle is available. We have to assume SOMETHING is working,
No question we have to assume something is workin -- just assume,
everybody knows that.
The issue is whether it must be noweb. I think "no". But, I do
I doubt it has to be noweb - my preference would be for a lisp based tool, but
I can see arguments for bash, sed, perl, and other such tools. (Particularly
since a lisp distribution is a poor assumption to make on most computers in
the world today :-( .) Really, we should have a "rosetta" set of tools to
translate pamphlets to buildable files (e.g. the notangle step) in as wide a
variety of tools as possible, since that step is so critical and needs
something working to bootstrap it. I expect getting the source code of of a
pamphlet is probably simpler as a dedicated operation than all of noweb
(particularly with a universal, standard nomenclature denoting source code) -
maybe a collection of such scripts could replace the current content of the
boot directory once we don't need boot ;-). ('course, I should try writing
one before I conclude it's doable...) Then we could reduce the "something
working" requirement from noweb to "one of noweb, Lisp, perl, sed, bash, csh,
etc. etc. etc."

In a sense, one might also say that depending on LaTeX is a bit unfortunate to
produce human readable documentation, because a TeX installation is not
especially trivial (nor is it common on Windows). However, the highly
nontrivial nature of LaTeX and TeX makes that a much more difficult problem,
and I don't think we have a choice but to assume a working TeX installation
for the time being. cl-typesetting might eventually evolve into a viable
alternative we could build and run off of the same technology used to build
and run Axiom itself, but it's nowhere near ready yet. And we can't afford
the space bundling something like teTeX and MiKTeX (or even a subset of them)
would take.

Anyway. Interesting to think about, and maybe even practical, but that's for
later. Right now, THANK YOU Gabriel for making a configure/make/make install
routine for Axiom!!!

Cheers,
CY
Gabriel Dos Reis
2006-08-02 15:05:55 UTC
Cliff Yapp <***@yahoo.com> writes:

[...]

| > | In a way, it's the same problem Tim faced with Axiom being needed to
| > | build Axiom. The only way out was (and still is) to have some files in
| > | lisp present, to teach lisp to read and understand BOOT and SPAD. Same
| > | deal here.
| >
| > That is a good recipe for maintenance nightmarre -- recent debates about
| > redundant boot anyone? There ought to be a better way to deal with it.
|
| I think the way out is clear - put all of the low level logic in Lisp, and the
| high level logic in Aldor (or SPAD, until Aldor is available).

Unfortunately, I don't have an infinite amount of time to wait. The
time I'm investing now, I'm expecting it to pay off when I tell
students "go and fetch Axiom" and "follow standard installation", or
when I eventually succeed in convincing some linux distros to have

[...]

| Anyway. Interesting to think about, and maybe even practical, but
| that's for later.

Yes.

| Right now, THANK YOU Gabriel for making a configure/make/make install
| routine for Axiom!!!

You're welcome.

-- Gaby
Ralf Hemmecke
2006-08-02 21:38:02 UTC
Post by Gabriel Dos Reis
| If I understand the question, you are asking how we can have only literate
| program files in Axiom at the get go and still not depend on noweb. The
| simple answer is, we can't.
Actually, we can if we don't insist that literate files must be
pamphlet files or we must use noweb :-)
Well, isn't it much simpler? We are speaking here about the generation
of code, so we mean notangle.

If for the bootstrapping file configure.ac.pamplet (that should be
enough?) we restrict to just several code chunks of the form

<<*>>=
code here
@
...
<<*>>=
other code
@

All these <<*>> chunks are linearly concatenated by notangle to give the
final output. For that simple task, I am sure that it is not too
difficult to write a sed or awk script. And the initial dependency on
noweb would be gone. Any sed/awk-programmers out there? Ehm, but then it
might depend on awk... what programs are usually assumed to exist before
I run ./configure? Windows/Un*x-like/MacOS systems are quite differnt.

I am not sure whether also the Makefile.am.pamphlet files could/should
be written in such a linear way. But all the non-pamphlet world can
handle just Makefile.am and that is linear. So where would a pamphlet
format impose some dependency on noweb for the configure process?

Ralf
Gabriel Dos Reis
2006-08-02 23:51:59 UTC
Ralf Hemmecke <***@hemmecke.de> writes:

| > | If I understand the question, you are asking how we can have only
| > literate | program files in Axiom at the get go and still not depend
| > on noweb. The | simple answer is, we can't. Actually, we can if we
| > don't insist that literate files must be
| > pamphlet files or we must use noweb :-)
|
| Well, isn't it much simpler? We are speaking here about the generation
| of code, so we mean notangle.

We're speaking of the bootstrapping process. We need to start
from somewhere. I believe notangle is not a wrong starting point.

But as I said, I implemented the majority's view. I just disagree
with it. My disagreement does nto change what end user will have to
type to install Axiom, when the project is completed.

| If for the bootstrapping file configure.ac.pamplet (that should be
| enough?) we restrict to just several code chunks of the form
|
| <<*>>=
| code here
| @
| ...
| <<*>>=
| other code
| @
|
| All these <<*>> chunks are linearly concatenated by notangle to give
| the final output.

Yes, that is what I implemented in configure.ac.pamphlet.

| For that simple task, I am sure that it is not too
| difficult to write a sed or awk script.

If you have that script, I'll drop my objection -- because it means
our bootstrapping dependency is much weaker, i.e. only traditional
tools.

| And the initial dependency on noweb would be gone.

Exactly.

| Any sed/awk-programmers out there? Ehm, but then it might depend on
| awk...

| what programs are usually assumed to exist
| before I run ./configure?

That is the next round of patches I'll send. Currently, there are
lots of unspoken dependencies deeply hidden in the makefiles.

| Windows/Un*x-like/MacOS systems are quite differnt.

Oh yeah, but I don't worry too much for the MacOS system as the
Unix-like enough.

| I am not sure whether also the Makefile.am.pamphlet files could/should
| be written in such a linear way. But all the non-pamphlet world can
| handle just Makefile.am and that is linear. So where would a pamphlet
| format impose some dependency on noweb for the configure process?

Makefile.am does not need to be linear. In fact, Makefile.am is
sufficiently high-level enough that if and when we get there, we will
see that Tim already did the job. We just need to re-structure
Makefile.pamphlet first. You won't have to write Makefile.am,
because you will write Makefile.pamphlet (should they be renamed to
Makefile.am.pamphlet? I don't know). Then you get Makefile.am out of
it by build-setup.sh. Once the system is in place, I do expect very
little modifications of makefiles; I suspect that 95% contribution
will be about algebra codes, 2% about the compiler... So, the current
project is not a big deal.

-- Gaby
Ralf Hemmecke
2006-08-03 08:13:17 UTC
Hi Gaby,
Post by Gabriel Dos Reis
| If for the bootstrapping file configure.ac.pamplet (that should be
| enough?) we restrict to just several code chunks of the form
|
| <<*>>=
| code here
| ...
| <<*>>=
| other code
|
| All these <<*>> chunks are linearly concatenated by notangle to give
| the final output.
Yes, that is what I implemented in configure.ac.pamphlet.
| For that simple task, I am sure that it is not too
| difficult to write a sed or awk script.
If you have that script, I'll drop my objection -- because it means
our bootstrapping dependency is much weaker, i.e. only traditional
tools.
Last night I saw that Automake depends on Perl. I am much more fluent in
Perl. And Perl runs und Window, too. I believe a simple notangle
replacement for the configure files would be much easier in Perl. Do you
allow me to write it in Perl? Or is that dependency already too strong?

Ralf
Gabriel Dos Reis
2006-08-03 10:19:40 UTC
Ralf Hemmecke <***@hemmecke.de> writes:

| Hi Gaby,
|
| > | If for the bootstrapping file configure.ac.pamplet (that should be
| > | enough?) we restrict to just several code chunks of the form
| > | | <<*>>=
| > | code here
| > | @
| > | ...
| > | <<*>>=
| > | other code
| > | @
| > | | All these <<*>> chunks are linearly concatenated by notangle to
| > give
| > | the final output.
| > Yes, that is what I implemented in configure.ac.pamphlet.
| > | For that simple task, I am sure that it is not too
| > | difficult to write a sed or awk script. If you have that script,
| > I'll drop my objection -- because it means
| > our bootstrapping dependency is much weaker, i.e. only traditional
| > tools.
|
| Last night I saw that Automake depends on Perl. I am much more fluent
| in Perl. And Perl runs und Window, too. I believe a simple notangle
| replacement for the configure files would be much easier in Perl. Do
| you allow me to write it in Perl? Or is that dependency already too
| strong?

That would be a good thing, at least.

-- Gaby
Ralf Hemmecke
2006-08-03 08:34:54 UTC
Post by Gabriel Dos Reis
Makefile.am does not need to be linear. In fact, Makefile.am is
sufficiently high-level enough that if and when we get there, we will
see that Tim already did the job. We just need to re-structure
Makefile.pamphlet first. You won't have to write Makefile.am,
because you will write Makefile.pamphlet (should they be renamed to
Makefile.am.pamphlet? I don't know).
I guess the literate idea even says that it does not matter how a file
is called. It is most important that you write a paper from which you
can generate all the code (even different files from one pamphlet
source). That sounds nice, but in some sense I find that very difficult
to maintain. For ALLPROSE I set the convention every file is a .nw file.
(I don't yet had the need to store file types like .fig or .png or other
strange formats where the noweb-style makes no sense.)
The file keeps its original name with just .nw added.
root
2006-08-03 14:10:31 UTC
Post by Ralf Hemmecke
I guess the literate idea even says that it does not matter how a file
is called. It is most important that you write a paper from which you
can generate all the code (even different files from one pamphlet
source). That sounds nice, but in some sense I find that very difficult
to maintain. For ALLPROSE I set the convention every file is a .nw file.
I suggest to adopt that convention for axiom (replace .nw by .pamphlet).
Ralf,

In fact the system direction is the exact opposite, at least the parts
I touch. C (and, even more egregious, Java) adopt the file-per-function
kinds of idea.

If you look back in history (or lived it as I have), you'll see that the
tiny-file idea is based on computer restrictions. Memory was small and
the compiler couldn't handle large files. Furthermore, the loader couldn't
load large files. So everyone uses small files (or, in my case, short
paper tapes).

You see this legacy everywhere. It is still possible to write memory
overlays in linkers so that when one function completes it can be
relaced in memory by another link section. This allows a large program
to run in small physical memory. This "feature" is still supported by
the intel chip with segmented memory (which is rarely used but still
causes pain). Limits of storage, compiler technology and imagination
caused the idea of splitting out header files as a way to expose
signatures. Now we live in header-file hell. "Libraries" were ways
to collect all the little pieces so the linker could connect them
and now we have DLL hell and lib* hell. Indeed we are forced to use
"IDE"s because the tiny files have overwhelmed our ability to cope.

The #line directive is a hack to allow a runtime to figure out which
grain-of-sand-file contains this particular function. It is also a
legacy of history. I am building a program for work that lives in one
literate file, has 30,000 lines of Lisp code, and has 2000+ pages of
final pdf documentation (so far) and 7000 test cases. It is never
ambiguous which file contains the failing function.

Computer programs have nothing to do with their file storage but
we have linked these ideas and suffered for it. Suppose we follow
the past and make Axiom "include" files which separate out the
signature information in a domain or category into separate files.

Consider what happens if you scale Axiom by a factor of 100 in
the next 30 years onto a petamachine. You end up with 110,000
domains and categories with roughly 1,100,000 functions. I don't
want 110K files. I want 110 books.

I have a library that must include 20 books on group theory.
They each have a section on permutation groups. It would be more
"efficient" if there were a standardized "permutation group" booklet
that everyone referenced as "included" into their text. That way there
would be no duplication. But that misses the point of literate works.

The key point is not that I talk about permutation groups but that I
talk about them in a certain way, in a certain place, with a certain
style, for a certain audience. The fact that the information is
available elsewhere means nothing to the human.

And we're trying to write for humans, not for machines.

I want to send you one book that you can just sit and read.
And, as a side-effect, you can also compile and run it. The running
program illustrates the ideas in a way that you can build on them.
You can modify them, extend them, write about them, and distribute
them, solve problems with them. You can build on other people's work.

We need to lift our eyes back to the humans, away from the technology.
Communicate, don't program.

Tim
Ralf Hemmecke
2006-08-03 18:33:55 UTC
Post by root
Post by Ralf Hemmecke
I guess the literate idea even says that it does not matter how a file
is called. It is most important that you write a paper from which you
can generate all the code (even different files from one pamphlet
source). That sounds nice, but in some sense I find that very difficult
to maintain. For ALLPROSE I set the convention every file is a .nw file.
I suggest to adopt that convention for axiom (replace .nw by .pamphlet).
Ralf,
In fact the system direction is the exact opposite, at least the parts
I touch. C (and, even more egregious, Java) adopt the file-per-function
kinds of idea.
You won't believe, but I agree. The literate idea contradicts Java's
one-file-per-class.

But what I think is important is, not just to write HUGE files that
nobody can manage anymore, but to add a bit of structure to it. Nowadays
we could write a whole book into just one latex file. But I bet, most
people structure that via several files to keep things manageable (in
the head not because of the file system).

We should also note that we are not yet at a point where we can really
start to work on the mathematics that we all love to do. We are
struggeling with the building process. So my suggestion/convention is
just to keep the build simple. If you come up with a better one, welcome.

I think any convention is better then the mess we currently have. What I
have seen in the Axiom build process is one-pamphlet-per-Makefile. That
is very much JAVA-like. One would have ONE document that describes the
whole build process, that is clear. But that document is a generated one
(dvi/pdf/...) but not necessarily just ONE pamphlet.

If you have the dvi, click on it and your editor jumps directly to the
appropriate source, you would not even care how the file is called. But
if you impose a certain structure on the files (like the
Makefile.am.pamphlet structure that Gaby is going to create), the whole
build process can be described a lot easier, since it relies on the
standard GNU Autotools. If you put all the various Makefile.am files
into just ONE big pamphlet, you even need a description (and some shell
script) that extracts all those different files from your .pamphlet.
Note that you would have to call notangle several times.

If everyone adopts his own conventions of what files are in the
pamphlet, that basically means that everyone provides his own (certainly
non-standard) scripts to transform the pamphlet into actual source code.
I don't think I would call that manageable, even if the pamphlet
describes what it does. Conventions/Rules are the way to go.

One rule we already have: Everything should be a pamphlet.
Next rule should be that there has to be a certain structure of the
pamphlet so that a number of the pamphlets can be used to produce one
document (a book if you like) that describes the build, another set of
pamphlets (that need not be disjoint from the first one) that describes
the interpreter etc. (But the "certain" above is still unclear, at least
I want that there is no \documentclass, \begin{document}, \end{document}
anymore.--The \usepackage is a problem, I know and have not yet a good
solution.)
Post by root
Indeed we are forced to use "IDE"s because the tiny files have
overwhelmed our ability to cope.
If you think that IDE's are just for that purpose, I think you missed
something. You get more than that. For example, if your mouse moves over
a function, you see its API description. You could click and jump to the
definition of the function. Right, that sounds like programming, but
even if we write pamphlets we finally have to write some code so if your
IDE would bring you immediately to the section where the function that
you want to use is described/defined, that would be convenient and make
your work more productive. It is just another workflow you have to
become accustomed to.

And somehow you have to turn all the codepieces into an actual source
code of a certain programming language. Pamphlets are nice, but it is a
bit hard to debug the code that comes out. You know, we are all not
perfect, so debugging is still an issue.
Post by root
The #line directive is a hack to allow a runtime to figure out which
grain-of-sand-file contains this particular function. It is also a
legacy of history. I am building a program for work that lives in one
literate file, has 30,000 lines of Lisp code, and has 2000+ pages of
final pdf documentation (so far) and 7000 test cases. It is never
ambiguous which file contains the failing function.
OK, an example: My Aldor program tells me that there is an error in
the + function. How long would it take you to find the right function
definition without a compiler telling you a line number, if you have 100
definitions of + in you 10,000 lines pamphlet? Even having just 10
definitions of +. It's a waste of time if I am looking at the wrong
place in 90% of the cases.
Post by root
Computer programs have nothing to do with their file storage but
we have linked these ideas and suffered for it. Suppose we follow
the past and make Axiom "include" files which separate out the
signature information in a domain or category into separate files.
If I use a category like "Ring" I am surely NOT going to "include" a
definition of "Ring" into my code. It is completely enough for me if the
"Ring" that appears in my code+description is a hyperlink and leads me
to the right place in the generated DVI/PDF/HTML. Especially if you
think of the web as a big document, why would you want to include and
reproduce all existing things again and again?
Post by root
Consider what happens if you scale Axiom by a factor of 100 in
the next 30 years onto a petamachine. You end up with 110,000
domains and categories with roughly 1,100,000 functions. I don't
want 110K files. I want 110 books.
Yes. But if your editor would not even tell you how many different files
your document consists of, why would you care whether you have 110K or
220K or just 110 files? We don't yet have the right tools to abstract
from the file system.
Post by root
We need to lift our eyes back to the humans, away from the technology.
Communicate, don't program.
I support this idea. But somebody has to provide the underlying
technology, right? And Axiom partly is or will become such a system. You
don't want to start from the very beginning, but you want to draw a
line. You don't explain how a computer is built if you want to speak

And the "book" idea is good, but it is not perfect. You have already
said that you want to write books for different people, look from a
different perspective onto the same data. So wouldn't it be nice to have
all these data (= some information) split into atoms that could be put
basically have such an idea of different view onto the same data.

And actually, if we used our Wiki more extensively, also that could
provide different views onto Axiom. I think, even in 20 years the 30
years horizon is still that far away.

Ralf
Page, Bill
2006-08-02 03:20:30 UTC
Post by Cliff Yapp
...
1) Have configure.ac and configure.ac.pamphlet both present
in the top level directory. configure.ac.pamphlet is where
editing is done, and when a commit is made (for this one file)
a configure.ac file is also committed
...
I don't see why we need 'configure.ac'. Just 'configure' and
configure.ac.pamphlet should be enough. No? It is not so
different than the fact that we have both Makefile and
Makefile.pamphlet now in the current Axiom distribution. With
the new configure process this "bootstrap" just moves back
one step.

As I understand it configure.ac would only be necessary in
order to generate a new 'configure' file and this would
normally only be done by an ambitious developer *after* they
had successfully installed Axiom at least once.
Post by Cliff Yapp
2) Extracting the documentation out of configure.ac.pamphlet
would require the use of the notangle command ...
Actually the 'noweave' command extracts the LaTeX file. 'notangle'
is required to extract configure.ac from configure.ac.pamphlet.
They are both parts of noweb which is either pre-installed or
optionally installed as one of the first steps in the axiom
build process.

The possibility of using some other literate programming tool
besides noweb is an interesting issue, but I think we should
agree that it is beyond the scope of the work.
Post by Cliff Yapp
...
I don't think we should fear a couple of non-pamphlet files
being present in the top level directory - indeed, it will
simplify matters considerably from the user standpoint.
I agree. As a user I would like to see just the usual:

./configure
make
make test
make install
axiom

What happens inside I don't care (unless it fails).

If ./configure says:

noweb is missing. Please install noweb first or specify
the option --with-noweb

I think most users of open source who are willing to build
from source would be able to handle that.

I think Axiom developers can live with just a little more
complication than that in order to handle the literate
programming methodology. So they need to have noweb installed.
built Axiom successfully. In that case they can use notangle
to extract a modified configure.ac from an edited version of
configure.ac.pamphlet before running automake/autoconf to
generate the new 'configure' and Makefile.in for a revised
distribution.

To be nice to new developers we might want to provide

make distribution

to automate this process.
Post by Cliff Yapp
configure.ac
buildaxiom.sh
buildaxiom_windows.<whatever windows uses>
buildXXX.sh is not so common. I would prefer to see just the
following pre-extracted as read-only files in the root of the
distribution:

configure
Makefile.in

with all others just .pamphlet files. The three files above
would also be contained as chunks in one of the .pamphlet
files - probably Makefile.pamphlet where all the other build
documentation is maintained. Because the Axiom build uses
recursive make on subdirectories, perhaps it would be convenient
if the Makefile.in files were present on each sub-directory
of the distribution. But as usual it is just the pamphlet
files that would be edited.

Regards,
Bill Page.
Gabriel Dos Reis
2006-08-02 03:45:11 UTC
"Page, Bill" <***@drdc-rddc.gc.ca> writes:

| On Tuesday, August 01, 2006 10:12 PM Cliff Yapp wrote:
| > ...
| > I thought it would end up like this:
| >
| > 1) Have configure.ac and configure.ac.pamphlet both present
| > in the top level directory. configure.ac.pamphlet is where
| > editing is done, and when a commit is made (for this one file)
| > a configure.ac file is also committed
| > ...
|
| I don't see why we need 'configure.ac'. Just 'configure' and
| configure.ac.pamphlet should be enough. No?

Autoconf needs an input to generate configure, that ought to be
configure.ac. You can imagine that the configure.ac exists only for
the purpose of generating configure, but things are not that simple
when one uses to other tools that depend on configure.ac.

[...]

| > I don't think we should fear a couple of non-pamphlet files
| > being present in the top level directory - indeed, it will
| > simplify matters considerably from the user standpoint.
|
| I agree. As a user I would like to see just the usual:
|
| ./configure
| make
| make test
| make install
| axiom
|
| What happens inside I don't care (unless it fails).

Yes.

[...]

| I think Axiom developers can live with just a little more
| complication than that in order to handle the literate
| programming methodology. So they need to have noweb installed.
| This would certainly be the case anyway if they had already
| built Axiom successfully.

Not necessarily. If I'm packaging axiom for a system, I'll not
necessarily have nowed installed, even after I built Axiom.

[...]

| To be nice to new developers we might want to provide
|
| make distribution

When we move (later) to Automake, we will have

make dist

-- Gaby
Page, Bill
2006-08-02 03:48:48 UTC
Gaby,
Post by Gabriel Dos Reis
...
I'm not a windows developer, so I'll leave the windows stuff
to others (hi Bill).
The windows build uses MSYS/MingGW so there is not much
difference from the usual linux build process. All of the
standard developer tools work in (almost) the same way.

The only complication here about Axiom on windows is that the
patches to the original axiom--main--1 linux distribution made
nearly two years ago by Mike Thomas to create axiom--windows--1
have not yet been merged back into the main sources. This is
not such a big deal - but somebody has to do it ... :)

A new release of Axiom on windows is long over due. Is anyone
interested in taking on the task? I am available to help but
not to lead the effort. And of course there is a chance that
we might be able to get Mike involved again. :)

Regards,
Bill Page.
root
2006-08-02 04:10:35 UTC
Bill,

Do you have an up-to-date set of sources from which you can build the
windows image? Are they someplace I can get at them?

t
Page, Bill
2006-08-02 04:59:10 UTC
Tim,
Post by root
Do you have an up-to-date set of sources from which you can
build the windows image? Are they someplace I can get at them?
Up-to-date? No, only the originals in the arch axiom--windows--1
branch and the same sources in darcs - both dated about Tue,
15 Mar 2005. :(

You can get to either of these for windows (but using tla on
Windows is a bit awkward). For details see:

"How I Built Axiom on Windows" at

http://wiki.axiom-developer.org/BuildAxiom

The only things missing here are the list of gcc files that must
be included in order to compile SPAD code and how to use NSYS to
build the actualy binary distribution for windows. If anyone gets
that far, I can help fill in these gaps.

Regards,
Bill Page.
Page, Bill
2006-08-03 19:37:03 UTC
Post by Ralf Hemmecke
...
And the "book" idea is good, but it is not perfect. You have
already said that you want to write books for different people,
look from a different perspective onto the same data. So wouldn't
it be nice to have all these data (= some information) split into
atoms that could be put together to new molecules (=books)? Read
about the LEO system, they basically have such an idea of
different view onto the same data.
+1 For sure. Leo is a "new generation" literaterate programming
tool that is well worth looking into. I think we would learn
a lot about Axiom if we were to set up a Leo environment of
building Axiom.
Post by Ralf Hemmecke
And actually, if we used our Wiki more extensively, also that
could provide different views onto Axiom. I think, even in 20
years the 30 years horizon is still that far away.
++1 Double for sure! :) I would very much like to continue to
explore this possibility :( if I ever find some more time ):

To put it crudely in the vernacular of the current generation:

"We don't need no stink'n 2000 page book! We're web connected."

Seriously, I think the web (the whole web, not just what fits
on your desktop) is the only technology that can solve this
sort of problem at the "peta-scale" that, as Tim has suggested,
is nearlly here already. Unfortunately we are only just being
to learn how to build such systems.

Regards,
Bill Page.
C Y
2006-08-03 21:05:57 UTC
Post by Ralf Hemmecke
But what I think is important is, not just to write HUGE files that
nobody can manage anymore, but to add a bit of structure to it.
Nowadays we could write a whole book into just one latex file. But
I bet, most people structure that via several files to keep things
manageable (in the head not because of the file system).
Bingo. When I started trying to write Maximabook way back when, one of
the first things I did was learn how to combine individual files using
a "master" control document file. Even if the editor can handle tens
of thousands of lines (and syntax highlighting those lines!) I can't
parse that much information without help.

I prefer the idea of "one file per concept", which happens to map
fairly well to the journal/academic paper model if we want an Axiom
journal. I envision the algebra structure as being composed of "core"
files which implement the basic foundations the system is built on and
most basic functionality (e.g. the current pamphlets) and then seeing
the system gradually expand in two ways. 1) individual papers
(pamphlets) contributed via journal or individual authors having to do
with specific subjects 2) as the quality of individual contributions
is proven and they become more central to the functionality being
implemented, fold the individual paper or papers that are becoming
"core" into the main file pertaining to that subject (so in a core
file, "chapters" or maybe "sections" would have their own subject
categorization. Or, using MSC, we have one file for each "top-level"
category that implements the standard functionality of Axiom for that
mathematical domain, and papers in sub-topics that address more
specific areas.

This could also address a concern I have with the wiki and "open"
nature of Axiom - namely, that Axiom is not really a "peer-reviewed"
work. Normal open source is "potentially" peer-reviewed, meaning
anybody can review it but it is not guaranteed that it has been
reviewed. The wiki content, by design, is even less guaranteed to be
"correct" in any academically verified sense. This could pose a
problem with being accepted for use in academic work - e.g. the lack of
authority behind the conceptual correctness of the work in question.
The question arises - "is this functionality I'm using written by
someone who knows what he/she is doing, and/or has it been checked and
verified by someone who can recognize when it's working incorrectly?"
The "core" files in each subject will be Axiom's "branded"
functionality - the functionality that we as a project stand behind as
being high quality. There are a few other ideas that need exploring
here, like systematic peer review of functionality, but I think it is
an important question that will bear on the acceptance of Axiom into
the academic community in the future. Wikipedia faces the same issue -
how do you know what you're reading is written by someone who knows
what they are doing, and doesn't just represent "common wisdom?"
Post by Ralf Hemmecke
We should also note that we are not yet at a point where we can
really start to work on the mathematics that we all love to do. We
are struggeling with the building process. So my
suggestion/convention is just to keep the build simple. If you come
up with a better one, welcome.
I think we are moving in that direction. Axiom is a complex system, so
there are certain limitations to the build simplicity, but I am very
glad to see a configure/make/make install routine arrive - I think that
is a very important first step.
Post by Ralf Hemmecke
I think any convention is better then the mess we currently have.
What I have seen in the Axiom build process is one-pamphlet-per-
Makefile. That is very much JAVA-like. One would have ONE document
that describes the whole build process, that is clear. But that
document is a generated one (dvi/pdf/...) but not necessarily just
ONE pamphlet.
As I understand it, the autotools will eventually get us close to that
goal. I think we won't be able to reduce it beyond an autotools build
process triggering a lisp based build process, but that IMHO is the
best possible situation. Maxima was brought to that point some time
ago and the system has been remarkably successful - I think it shows
"the way forward" for large free lisp programs in general. Autotools
interfaces with the "normal" software development world (as well as
handling a score of practical points we don't want to worry about), and
lets us keeps the "lisp and beyond" parts in Lisp, which has a number
of advantages from a development standpoint. Autotools +
defsystem/asdf is a very powerful and flexible setup.
Post by Ralf Hemmecke
One rule we already have: Everything should be a pamphlet.
Next rule should be that there has to be a certain structure of the
pamphlet so that a number of the pamphlets can be used to produce one
document (a book if you like) that describes the build, another set
of pamphlets (that need not be disjoint from the first one) that
describes the interpreter etc. (But the "certain" above is still
unclear, at least I want that there is no \documentclass, \begin
{document}, \end{document} anymore.--The \usepackage is a problem,
I know and have not yet a good solution.)
I experimented with this some time ago, actually - having the actual
content be in a "content-only" latex file, and having two types of
"top-level" files - one that pulled several "content-only" files into a
book, and the other type (a standard file for all pamphlet files which
could be altered on an individual file basis) which would let the user
print an individual "paper". IIRC there wasn't much interest in this
at the time and I abandoned the experiment.
Post by Ralf Hemmecke
If I use a category like "Ring" I am surely NOT going to "include" a
definition of "Ring" into my code. It is completely enough for me if
the "Ring" that appears in my code+description is a hyperlink and
leads me to the right place in the generated DVI/PDF/HTML.
Especially if you think of the web as a big document, why would you
want to include and reproduce all existing things again and again?
One illustration of presenting information in different ways might be
illustrated by the dimensions example. In the units paper, I need to
explain several programming concepts "polymorphism" in order to make
sense of how to express dimensional ideas in a computer algebra system.
Now, if we make all of Axiom fully literate there will undoubtedly be
a "proper" explanation of polymorphism somewhere in the compiler part
of the software, but I'll want to explain it with respect to the units
code because someone reading up on units is probably doing it for
scientific work in the physical sciences. Thus, an explanation of
polymorphism for a programmer might be less useful to a physical
scientist than the explanation targeted for that particular audience.

However, I agree that in general a link to an explanation is preferable
- we should avoid information duplication which does provide a real
benefit to the end user.

Cheers,
CY

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Gabriel Dos Reis
2006-08-03 21:16:04 UTC
C Y <***@yahoo.com> writes:

[...]

| I prefer the idea of "one file per concept",

strongly agreed.

For me, the issue here is that of organizing information for human
comprehension. I simply cannot handle gigantic file, no matter how
powerful my computers are.

-- Gaby