Local Development from Scratch

Requirements

Make sure you have these things installed on your system:
  • Git
  • Python 3.8.x
    • python3-venv (to setup virtual enviroment)
    • python3-pip (to install python packages)
  • PostgreSQL 12.x
    • libpq-dev (on Linux at least)
  • Apache or Nginx
  • Node 12.x
On Linux install them with your normal package manager. On macOS Homebrew is an excellent option.
For Windows Chocolatey seems popular but we have no experience with Windows.

Domains for local development

You will need two domain to run this app. One for the public site and one for the apply site.
Add this to your /etc/hosts file. (Feel free to use another name but then remember to use it in all the commands below.)
1
127.0.0.1 hypha.test
2
127.0.0.1 apply.hypha.test
Copied!
The "test" TLD is safe to use, it's reserved for testing purposes.
OBS! All examples from now on will use the hypha.test domains.

Get the code

1
$ git clone https://github.com/HyphaApp/hypha.git hypha
2
3
$ cd hypha
Copied!
OBS! Everything from now on will happen inside the hypha directory.

Python virtual environment

Create the virtual environment, specify the python binary to use and the directory. Then source the activate script to activate the virtual environment. The last line tells Django what settings to use.
1
$ python3 -m venv venv/hypha
2
$ source venv/hypha/bin/activate
3
$ export DJANGO_SETTINGS_MODULE=hypha.settings.dev
Copied!
Inside your activated virtual environment you will use plain python and pip commands. Everything inside the virtual environment is python 3 since we specified that when we created it.
Each time you open up a new shell to work with the app you will need to activate the virtual environment.
1
$ cd /path/to/application/hypha
2
$ source venv/hypha/bin/activate
3
$ export DJANGO_SETTINGS_MODULE=hypha.settings.dev
Copied!

Install Python packages

All the needed python packages for production are listed in the requirements.txt file. Additional packages for development are listed in requirements-dev.txt, it also includes everything from the requirements.txt file.
For a development environment you then run:
1
$ pip install -r requirements-dev.txt
Copied!
If any requirements*.txt file have been updated you will need to rerun this command to get the updated/added packages.
Note for MacOS users: On more recent systems like Mojave, you may need to run sudo installer -pkg /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg -target / before installing with pip for psycopg2 to install correctly.

Install Node packages

All the needed Node packages are listed in package.json. Install them with this command.
1
$ npm install
Copied!
You will also need the gulp task manager. On some systems you might need to run this command with sudo.
1
$ npm install -g gulp-cli
Copied!

The Postgres database

If the createdband dropdb is not available you will need to add the Postgres bin directory to your path or call the commands with complete path.
Create a database for the app.
1
$ createdb hypha
Copied!
To drop a database use.
1
$ dropdb hypha
Copied!

Linux installs might require setting up a user

On Linux you might need to run as the "postgres" user first when setting up Postgres. Use it to create the database and set up a database user. For local development I suggest creating a user with the same name as your account, then you will not need to specify it on every command.
1
$ su - postgres
2
$ createdb hypha
3
$ createuser [your-account-name]
Copied!

macOS users might need this fix

To make the app find the Postgres socket you might need to update the "unix_socket_directories" setting in the postgresql.conf file.
1
unix_socket_directories = '/tmp, /var/pgsql_socket'
Copied!

Use stellar for db snapshots

If you installed "stellar" you can use it to take snapshots and restore them.
1
$ stellar snapshot hypha_2019-10-01
Copied!
1
$ stellar restore hypha_2019-10-01
Copied!

Local settings

On production it's recommended to use environment variables for all settings. For local development putting them in a file is however convenient.
When you use the "dev" settings it will included all the setting you put in local.py.
Copy the local settings example file.
1
$ cp -p hypha/settings/local.py.example hypha/settings/local.py
Copied!
You most likely want to set these:
  • ALLOWED_HOSTS
  • BASE_URL
  • SECRET_KEY
If you have a problem with "CSRF cookie not set".
1
CSRF_COOKIE_SAMESITE = None
2
SESSION_COOKIE_SAMESITE = None
Copied!
If you do not use the app name for the database.
1
import dj_database_url
2
3
DATABASES = {
4
'default': dj_database_url.config(
5
conn_max_age=600,
6
default='postgres:///your_db_name'
7
)
8
}
Copied!

Create log directory

mkdir -p var/log/

Set up Gunicorn

Gunicorn is installed inside the virtual environment since it's listed in requirements.txt.
At the bottom of this file you find a handy script for Gunicorn. For a quick test you can run this command from your site directory.
1
$ gunicorn hypha.wsgi:application --env DJANGO_SETTINGS_MODULE=hypha.settings.dev --bind 127.0.0.1:9001
Copied!
Quit the server with ctrl-c when you done testing.
  • hypha.wsgi:application links it up to the app via the hypha/wsgi.py file.
  • --env … tells it what settings to use, for development your want "dev" settings.
  • --bind … makes it listen on localhost port 9001. Socket works as well but on macOS they have given me issues, while tcp connections always seems to just work.

Set up Apache or Nginx

Set up new vhost for Apache or Nginx. We let the web server handle static files and proxy everything else over to Gunicorn.
The examples uses port 80 for web server and port 9001 for WSGI, feel free to change that if needed. You maybe are already running other services on these ports or prefer to not run the web server as root and want a port above 1024.

Apache

1
<VirtualHost 127.0.0.1:80>
2
ServerName hypha.test
3
ServerAlias apply.hypha.test
4
DocumentRoot "/path/to/application/hypha/hypha"
5
6
7
Alias /media/ /path/to/application/hypha/media/
8
Alias /static/ /path/to/application/hypha/static/
9
Alias /static_src/ /path/to/application/hypha/hypha/static_src/
10
11
<Directory "/path/to/application/hypha/static">
12
Require all granted
13
</Directory>
14
15
<Directory "/path/to/application/hypha/media">
16
Require all granted
17
</Directory>
18
19
<Directory "/path/to/application/hypha/hypha/static_src">
20
Require all granted
21
</Directory>
22
23
<IfModule mod_proxy_http.c>
24
<Location "/">
25
ProxyPreserveHost On
26
ProxyPass http://127.0.0.1:9001/
27
ProxyPassReverse http://127.0.0.1:9001/
28
</Location>
29
30
<LocationMatch "^/(static|media|static_src)/">
31
ProxyPass !
32
</LocationMatch>
33
34
<LocationMatch "^/[\w-]+\.(ico|json|png|txt)quot;>
35
ProxyPass !
36
</LocationMatch>
37
</IfModule>
38
</VirtualHost>
Copied!

Nginx

1
server {
2
listen 80;
3
server_name hypha.test apply.hypha.test;
4
5
location /media/ {
6
alias /path/to/application/hypha/media/;
7
}
8
9
location /static/ {
10
alias /path/to/application/hypha/static/;
11
}
12
13
location / {
14
proxy_set_header Host $http_host;
15
proxy_set_header X-Real-IP $remote_addr;
16
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
17
proxy_set_header X-Forwarded-Proto $scheme;
18
proxy_pass http://127.0.0.1:9001/;
19
}
20
21
}
Copied!

Front end development

See the gulpfile.js file for a complete list of commands. Here are the most common in development.
This will watch all sass and js files for changes and build them with css maps. It will also run the "collecstatic" command, useful when running the site with a production server and not the built in dev server.
1
$ gulp watch
Copied!
If you are working on the React components then it may be worth just using one of the two following commands. They should do the same thing, but the npm command calls Webpack direct.
First you also need to set "API_BASE_URL" to the correct value.
1
$ export API_BASE_URL='http://apply.hypha.test/api'
Copied!
1
$ gulp watch:app
2
# OR
3
$ npm run webpack-watch
Copied!
To build the assets which get deployed, use the following. The deployment scripts will handle this, and the files should not be committed.
1
$ gulp build
Copied!

Finally, the app itself

Start by specifying what settings file is to be used (if you have not already done this, see above).
1
$ export DJANGO_SETTINGS_MODULE=hypha.settings.dev
Copied!
Then decide if you want to start with some demo content or with an empty db.

Load the sandbox db to get some demo content

There is a /public/sandbox_db.dump file that has some demo content to get you started. Load it in with this command.
1
$ pg_restore --verbose --clean --if-exists --no-acl --no-owner --dbname=hypha public/sandbox_db.dump
Copied!
It's not always completely up to date so run:
1
$ python manage.py migrate --noinput
Copied!

Or create a db from scratch.

Create the cache tables.
1
$ python manage.py createcachetable
Copied!
Run all migrations to set up the database tables.
1
$ python manage.py migrate --noinput
Copied!
Create the first super user.
1
$ python manage.py createsuperuser
Copied!
Collect all the static files.
1
$ python manage.py collectstatic --noinput
Copied!
(If this command complain about missing static_compiled directory, run the gulp command above first.)
Set the addresses and ports of the two wagtail sites.
1
$ python manage.py wagtailsiteupdate hypha.test apply.hypha.test 80
Copied!
Now you should be able to access the sites on http://hypha.test/ and http://apply.hypha.test/

Run tests

Hypha has specific settings for testing so specify them when you run the "test" command.
1
$ python manage.py test --settings=hypha.settings.test
Copied!
If you need to rerun the tests several times this will speed them up considerably.
1
$ python manage.py test --parallel --keepdb --settings=hypha.settings.test
Copied!

Administration

Use the email address and password you set in the createsuperuser step above to login.

Useful things

Script to start/stop/restart gunicorn:
1
#!/usr/bin/env bash
2
3
set -euo pipefail
4
5
WORKERS=3
6
PORT=${2:-9001}
7
WSGI=$(find . -name wsgi.py -not -path "./venv/*")
8
WSGI=${WSGI%/*}
9
NAME=${WSGI##*/}
10
SETTINGS="${NAME}.settings.${3:-dev}"
11
RUNDIR="./var/run"
12
LOGDIR="./var/log"
13
SOCK="${RUNDIR}/${NAME}-gunicorn.sock"
14
PID="${RUNDIR}/${NAME}-gunicorn.pid"
15
LOGFILE="${LOGDIR}/${NAME}-gunicorn.log"
16
17
function start_gunicorn {
18
if [[ ! -d "${RUNDIR}" ]]; then
19
mkdir -p "${RUNDIR}"
20
else
21
rm -rf "${RUNDIR}/${NAME}*"
22
fi
23
24
if [[ ! -d "${LOGDIR}" ]]; then
25
mkdir -p "${LOGDIR}"
26
fi
27
28
exec gunicorn \
29
${NAME}.wsgi:application \
30
--env DJANGO_SETTINGS_MODULE=${SETTINGS} \
31
--pid ${PID} \
32
--bind 127.0.0.1:${PORT} \
33
--workers ${WORKERS} \
34
--name ${NAME} \
35
--daemon \
36
--reload \
37
--log-file=${LOGFILE} \
38
--log-level error
39
echo "Gunicorn started."
40
}
41
42
function stop_gunicorn {
43
kill `cat $PID`
44
echo "Gunicorn stopped."
45
}
46
47
case $1 in
48
start)
49
start_gunicorn
50
;;
51
stop)
52
stop_gunicorn
53
;;
54
restart)
55
stop_gunicorn
56
start_gunicorn
57
;;
58
*)
59
echo "Not a recognised command, use start/stop/restart."
60
;;
61
esac
Copied!
Last modified 7d ago