Setup the container#

You only need to run these setup steps once.

Requirements

You need to have Docker installed and running on your computer.
You need to have a GitHub account.

Fork repositories on GitHub#

Log in to your GitHub account.
Fork the following repositories:

geodynamics/pythia (optional, not recommended)
geodynamics/spatialdata (optional, not recommended)
geodynamics/pylith

This creates copies of the repositories in your GitHub account.

Create Docker volume for persistent storage#

On your local machine, create a Docker volume for persistent storage.

docker volume create pylith-dev

Start PyLith development Docker container#

Running the command below will:

Start (run) the Docker container using the pylith-devenv Docker image and assign it the name pylith-dev-workspace.
Mount the docker volume with persistent storage at /opt/pylith.
The pylith-devenv Docker image will be downloaded from the GitLab registry <registry.gitlab.com/cig-pylith/pylith_installer>.

docker run --name pylith-dev-workspace --rm -it -v pylith-dev:/opt/pylith \
    registry.gitlab.com/cig-pylith/pylith_installer/pylith-devenv

Note

The pylith-devenv Docker image is a multiple architecture image supporting both the amd64 (Intel) and arm64 (Apple M processor) architectures. Docker will automatically download the portion of the image appropriate for your architecture. In either case, the image uses the Linux Ubuntu 22.04 operating system.

Warning

Closing the pylith-dev-workspace Docker container interactive shell (terminal) will stop the container. Simply run the command again to restart the container.

Setup directory structure#

We will use the following directory structure for the persistent storage.

/opt/pylith
    ├── src
    │    ├── pythia
    │    ├── spatialdata
    │    ├── petsc
    │    └── pylith
    ├── build
    │   ├── pythia-debug
    │   ├── pythia-opt (optional)
    │   ├── spatialdata-debug
    │   ├── spatialdata-opt (optional)
    │   ├── pylith-debug
    │   └── pylith-opt (optional)
    ├── dest-debug
    │   ├── bin
    │   ├── include
    │   ├── lib
    │   └── share
    └── dest-opt (optional)
        ├── bin
        ├── include
        ├── lib
        └── share

All of the source code will be placed under /opt/pylith/src. You only need to create the top-level source directory as the subdirectories will be created when you clone (download) the repositories.

This directory structure is set up for both a debugging version for development (*-debug directories) and an optimized version for performance testing (*-opt directories). For now, we will only setup the debugging version.

cd ${DEV_DIR}
mkdir src
mkdir -p ${TOP_BUILDDIR} && pushd ${TOP_BUILDDIR} && mkdir pythia-debug spatialdata-debug pylith-debug && popd
mkdir -p ${INSTALL_DIR}

Clone repositories#

This creates a local copy of the repositories in the persistent storage volume of the PyLith development container. These are your working copies of the repositories.

cd ${TOP_SRCDIR}
git clone --recursive https://github.com/geodynamics/pythia.git
git clone --recursive https://github.com/geodynamics/spatialdata.git
git clone --recursive https://github.com/GITHUB_USERNAME/pylith.git
git clone --branch knepley/pylith https://gitlab.com/petsc/petsc.git

Set the upstream repository#

For the PyLith repository and any other repositories that you forked, you should set the upstream repository. The upstream repository is the central, community repository from which you will get updates. We also recommend never using the main branch in your fork; instead, set your local main branch to the upstream/main branch. This simplifies keeper your local repository in sync with the community repository.

# PyLith repository (repeat for other repositories you forked)
cd ${TOP_SRCDIR}/pylith
git remote add upstream https://github.com/geodynamics/pylith.git
git fetch upstream

# Checkout the `upstream` main branch
git checkout -b upstream/main --track upstream/main
# Remove the local `main` from your fork
git branch -D main
# Rename your `upstream/main` to `main`
git branch -m upstream/main main

Setup Python virtual environment#

We use a Python virtual environment, so that we can install Python modules using the Pip module. The Docker container explicitly sets the environment variables to make use of this virtual environment, so you do not need to manually activate it.

python3 -m venv ${PYLITH_DIR}

Configure and build PyLith for development#

We build 3 CIG-related dependencies and PyLith:

Pythia
Spatialdata
PETSc
PyLith

Pythia, Spatialdata, and PETSc are not include in the Docker image, because they need to be updated as part of PyLith development.

Tip

To speed up the build process, we set the number of make threads to the number of cores -j$(nproc). You can often find speedup with up to twice as many threads as the number of cores.

Pythia#

cd ${TOP_BUILDDIR}/pythia-debug
pushd ${TOP_SRCDIR}/pythia && autoreconf -if && popd
${TOP_SRCDIR}/pythia/configure --prefix=${PYLITH_DIR} --enable-testing \
    CC=mpicc CXX=mpicxx CFLAGS="-g -Wall" CXXFLAGS="-g -Wall"
make install
make check

Spatialdata#

cd ${TOP_BUILDDIR}/spatialdata-debug
pushd ${TOP_SRCDIR}/spatialdata && autoreconf -if && popd
${TOP_SRCDIR}/spatialdata/configure --prefix=${PYLITH_DIR} \
    --enable-swig --enable-testing \
	CPPFLAGS="-I${PYLITHDEPS_DIR}/include -I${PYLITH_DIR}/include" \
	LDFLAGS="-L${PYLITHDEPS_DIR}/lib -L${PYLITH_DIR}/lib --coverage" \
	CXX=mpicxx CXXFLAGS="-g -Wall --coverage"
make install -j$(nproc)
make check -j$(nproc)

PETSc#

cd ${TOP_SRCDIR}/petsc
python3 ./configure --with-c2html=0 --with-lgrind=0 --with-fc=0 \
    --with-x=0 --with-clanguage=C --with-mpicompilers=1 \
    --with-shared-libraries=1 --with-64-bit-points=1 --with-large-file-io=1 \
    --with-hdf5=1 --download-chaco=1 --download-ml=1 \
    --download-f2cblaslapack=1 --with-debugging=1 CFLAGS="-g -O -Wall" \
    CPPFLAGS="-I${HDF5_INCDIR} -I${PYLITHDEPS_DIR}/include" \
    LDFLAGS="-L${HDF5_LIBDIR} -L${PYLITHDEPS_DIR}/lib"
make 
make check

PyLith#

cd ${TOP_BUILDDIR}/pylith-debug
pushd ${TOP_SRCDIR}/pylith && autoreconf -if && popd
${TOP_SRCDIR}/pylith/configure --prefix=${PYLITH_DIR} \
    --enable-cubit --enable-hdf5 --enable-swig --enable-testing \
    --enable-test-coverage --with-python-coverage=coverage3 \
    CPPFLAGS="-I${HDF5_INCDIR} -I${PYLITHDEPS_DIR}/include -I${PYLITH_DIR}/include" \
    LDFLAGS="-L${HDF5_LIBDIR} -L${PYLITHDEPS_DIR}/lib -L${PYLITH_DIR}/lib --coverage" \
    CC=mpicc CFLAGS="-g -Wall" CXX=mpicxx CXXFLAGS="-g -Wall --coverage"
make install -j$(nproc)
make check -j$(nproc)

Reset `ptrace` Flag#

Tip

This step should no longer be needed. We are keeping it in the documentation for reference in case you are still using an older image.

Attaching the debugger to a forked process can result in the following error:

Could not attach to process. If your uid matches the uid of the target process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try again as the root user. For more details, see /etc/sysctl.d/10-ptrace.conf
ptrace: Operation not permitted.

The PyLith development environment Docker container sets up /etc/sysctl.d/10-ptrace.conf correctly, but the ptrace_scope variable is still usually 1. The fix is to run the container in privileged mode as root and restart the procps service.

# Run docker image in privileged mode as root.
docker run -ti --privileged --rm -u root registry.gitlab.com/cig-pylith/pylith_installer/pylith-devenv /bin/bash

# Verify ptrace setting needs updating
cat /proc/sys/kernel/yama/ptrace_scope
# If output is 1, then continue, if 0 then no need to change anything.

# Verify ptrace setting is correct.
cat /etc/sysctl.d/10-ptrace.conf
# Output should be 0

# Restart the procps service.
service procps restart

# Verify ptrace setting has changed
cat /proc/sys/kernel/yama/ptrace_scope
# Output should be 0

Install Visual Studio Code#

Install Visual Studio Code for your computer.
Install the following extensions:
- Dev Containers
- C/C++
- Docker
- Live Share
- Python
- Uncrustify
- Live Share
- C++ TestMate
- Test Explorer UI

We recommend also installing the following extensions:

autoconf
Code Spell Checker
Markdown all in One
markdownlint
Material Icon Theme
MyST-Markdown
Remote-SSH
Git Graph
Git Lens

Note

Git Lens provides a more substantial GUI to Git. If you are new to Git, you may find the GUI more intuitive than the command line. Git Lens has some add-on features that require subscriptions. If you find notifications for these add-ons annoying, you can easily disable or uninstall the Git Lens extension.

Setup the container#

Fork repositories on GitHub#

Create Docker volume for persistent storage#

Start PyLith development Docker container#

Setup directory structure#

Clone repositories#

Set the upstream repository#

Setup Python virtual environment#

Configure and build PyLith for development#

Pythia#

Spatialdata#

PETSc#

PyLith#

Reset ptrace Flag#

Install Visual Studio Code#

Reset `ptrace` Flag#