Gitlab Community Edition Instance

Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found
Select Git revision

Target

Select target project
  • hpc-team-public/synforest
1 result
Select Git revision
Show changes
Commits on Source (22)
Showing
with 16148 additions and 0 deletions
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
.pybuilder/
target/
# Jupyter Notebook
.ipynb_checkpoints
# IPython
profile_default/
ipython_config.py
# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version
# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock
# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock
# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
.pdm.toml
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/
# Celery stuff
celerybeat-schedule
celerybeat.pid
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
.dmypy.json
dmypy.json
# Pyre type checker
.pyre/
# pytype static type analyzer
.pytype/
# Cython debug symbols
cython_debug/
# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
# PyTreeDB
pytreedb
# Configs
config/*
# SynForest Data
data
# Container
container/*
# LASTOOLS
bin/LAStools
# Results
results/*
# Temp
tmp/*
\ No newline at end of file
# **SynForest**
## Point Cloud Generation for Large-Scale Synthetic Forests
SynForest is a tool that generates realistic large-scale point clouds of forests by simulating the lidar scanning process from a stationary or moving platform and capturing the shape and structure of realistic tree models.
![Sample forest](./docs/img/synthetic_forest_thick.png)
SynForest was employed to generate synthetic forest datasets that were utilized in model pre-training for instance segmentation in point clouds.
# 1. Installation
SynForest has multiple components that require preparation, so the installation involves several steps that we explain here.
## 1.1. SynForest
Clone this repository:
```bash
git clone https://github.com/adoosth/forest-synthesis.git
```
Create and activate the conda environment:
```bash
conda create -y -n synforest python=3.8
conda activate synforest
```
Install the `synforest` package:
```bash
cd forest-synthesis/synforest
pip install . -r requirements.txt
```
## 1.2. Container for Helios++
Download the container for Helios++ from [here](https://owncloud.gwdg.de/index.php/f/2417870773) (You may need to request access). SynForest expects the file `synforest.sif` to be placed under `container/`, but this can be changed in the configuration file if desired.
## 1.3. Lastools
If LAStools is not available already, you must obtain and compile it. The source code is available at: [https://github.com/LAStools/LAStools](https://github.com/LAStools/LAStools)
Clone the repository under `bin/`. You can also change the path later in the configuration file.
```git
git clone https://github.com/LAStools/LAStools.git
```
Now, compile LAStools:
```bash
cd LAStools
make
```
You should now have the exectuble file `las2las` located in `LAStools/bin/`:
```bash
./bin/las2las -h
```
## 1.4. PyTreeDB
PyTreeDB is a point cloud database consisting of real trees in Germany. These point clouds are used as base tree models for the process of forest generation.
A python script is provided to download the entire database with a compatible folder structure for use in SynForest. If you have enough space (at least 14 GB), it is recommended to store it under `pytreedb/`, otherwise you must set the path later in the configuration files.
```bash
python tools/get_pytreedb.py ./pytreedb
```
You may replace `./pytreedb` with the desired path to store PyTreeDB.
## 1.5. Configure Slurm
The tool depends on `slurm` for job submission. The partition names for the slurm jobs most likely need to be configured for your system. You may set the partition and other settings in the headers of the files `synforest/scripts/utils/sb_single_tile.sh` and `synforest/scripts/utils/sb_merge_tiles.sh` to pertain to your HPC system.
# 2. Usage
A number of sample configurations are provided in `config/`. To run a forest generation task, execute the `run.py` script with the path to the configuration file, i.e.:
```
cd synforest
python run.py config/example_ULS.json
```
If any of the components as described in Step 1 are located elsewhere, you must alter the configuration file accordingly.
For a full instruction manual of the configuration files, see [configuration](./docs/CONFIG.md).
# 3. Acknowledgements
- GWDG Computing Group: [Group Page](https://www.gwdg.de/web/guest/about-us/organization/departments/computing)
- HLRN: [https://hlrn.de](https://hlrn.de)
- ForestFactory: [https://git.ufz.de/angermue/forestfactory](https://git.ufz.de/angermue/forestfactory)
- Helios++: [https://github.com/3dgeo-heidelberg/helios](https://github.com/3dgeo-heidelberg/helios)
- Janika Schäfer et al. "<i>Applications of synthetic airborne laser scanning data of forests.</i>", ForestSat Conference 2022
# 4. Citation
If you used this tool in your research, please cite as follows:
```
@conference{synforest2023,
title = {Large-Scale Synthetic Forest Point Cloud Generation on HPC Systems},
author = {Ali Doosthosseini and Hauke Kirchner and Dorothea Sommer and Julian Kunkel},
year = 2023,
month = {September},
organization = {NHR Conference '23}
}
```
# 5. Contacts
If there are any issues or questions, feel free to contact Ali Doosthosseini [adoosth@gwdg.de](mailto:adoosth@gwdg.de) or Hauke Kirchner [hauke.kirchner@gwdg.de](mailto:hauke.kirchner@gwdg.de)
\ No newline at end of file
{
"id": "synforest-example-TLS-{timestamp}",
"work_dir": "data",
"map": {
"stand": "sample/sample_forest.res",
"side": 200,
"species": "QuePet"
},
"scene":{
"pytreedb_dir": "pytreedb",
"model_source": "TLS",
"min_quality": 3,
"has_leaves": true,
"tile_w": 50,
"tile_h": 50,
"tile_overlap": 10,
"template": {
"id": "{id}_{tile_id}",
"name": "{id}_{tile_id}",
"area": {
"z_var": 0.5
},
"trees": []
}
},
"simulation": {
"container": "container/synforest.sif",
"lastools": "bin/LAStools",
"output_dir": "results",
"template": {
"id": "{id}_{tile_id}",
"name": "{id}_{tile_id}",
"helios_seed": 42,
"platform": {
"id": "copter_linearpath",
"speed": 10,
"altitude": 80
},
"scanner": {
"id": "dji_zenmuse_l1",
"angle": 30.0,
"pulse_freq": 100000
},
"legs": [
{"x": 100, "y": 300},
{"x": 300, "y": 100},
{"x": 250, "y": 50},
{"x": 50, "y": 250},
{"x": 0, "y": 200},
{"x": 200, "y": 0},
{"x": 150, "y": -50},
{"x": -50, "y": 150},
{"x": -100, "y": 100},
{"x": 100, "y": -100},
{"x": 300, "y": 100},
{"x": 250, "y": 150},
{"x": 50, "y": -50},
{"x": 0, "y": 0},
{"x": 200, "y": 200},
{"x": 150, "y": 250},
{"x": -50, "y": 50},
{"x": -100, "y": 100},
{"x": 100, "y": 300}
]
}
}
}
\ No newline at end of file
{
"id": "synforest-example-ULS-{timestamp}",
"work_dir": "data",
"map": {
"stand": "sample/sample_forest.res",
"side": 200,
"species": "QuePet"
},
"scene":{
"pytreedb_dir": "pytreedb",
"model_source": "ULS",
"min_quality": 3,
"has_leaves": true,
"tile_w": 100,
"tile_h": 100,
"tile_overlap": 10,
"template": {
"id": "{id}_{tile_id}",
"name": "{id}_{tile_id}",
"area": {
"z_var": 0.5
},
"trees": []
}
},
"simulation": {
"container": "container/synforest.sif",
"lastools": "bin/LAStools",
"output_dir": "results",
"template": {
"id": "{id}_{tile_id}",
"name": "{id}_{tile_id}",
"helios_seed": 42,
"platform": {
"id": "copter_linearpath",
"speed": 10,
"altitude": 80
},
"scanner": {
"id": "dji_zenmuse_l1",
"angle": 30.0,
"pulse_freq": 100000
},
"legs": [
{"x": 100, "y": 300},
{"x": 300, "y": 100},
{"x": 250, "y": 50},
{"x": 50, "y": 250},
{"x": 0, "y": 200},
{"x": 200, "y": 0},
{"x": 150, "y": -50},
{"x": -50, "y": 150},
{"x": -100, "y": 100},
{"x": 100, "y": -100},
{"x": 300, "y": 100},
{"x": 250, "y": 150},
{"x": 50, "y": -50},
{"x": 0, "y": 0},
{"x": 200, "y": 200},
{"x": 150, "y": 250},
{"x": -50, "y": 50},
{"x": -100, "y": 100},
{"x": 100, "y": 300}
]
}
}
}
\ No newline at end of file
{
"id": "synforest-example-ULS-multi-{timestamp}",
"work_dir": "data",
"map": {
"stand": "sample/sample_forest.res",
"side": 200,
"species": ["PinSyl", "PicAbi", "FagSyl", "QueRub", "AceCam", "FraExc", "BetPen", "PseMen"]
},
"scene":{
"pytreedb_dir": "pytreedb",
"model_source": "ULS",
"min_quality": 3,
"has_leaves": true,
"tile_w": 100,
"tile_h": 100,
"tile_overlap": 10,
"template": {
"id": "{id}_{tile_id}",
"name": "{id}_{tile_id}",
"area": {
"z_var": 0.5
},
"trees": []
}
},
"simulation": {
"container": "container/synforest.sif",
"lastools": "bin/LAStools",
"output_dir": "results",
"template": {
"id": "{id}_{tile_id}",
"name": "{id}_{tile_id}",
"helios_seed": 42,
"platform": {
"id": "copter_linearpath",
"speed": 10,
"altitude": 80
},
"scanner": {
"id": "dji_zenmuse_l1",
"angle": 30.0,
"pulse_freq": 100000
},
"legs": [
{"x": 100, "y": 300},
{"x": 300, "y": 100},
{"x": 250, "y": 50},
{"x": 50, "y": 250},
{"x": 0, "y": 200},
{"x": 200, "y": 0},
{"x": 150, "y": -50},
{"x": -50, "y": 150},
{"x": -100, "y": 100},
{"x": 100, "y": -100},
{"x": 300, "y": 100},
{"x": 250, "y": 150},
{"x": 50, "y": -50},
{"x": 0, "y": 0},
{"x": 200, "y": 200},
{"x": 150, "y": 250},
{"x": -50, "y": 50},
{"x": -100, "y": 100},
{"x": 100, "y": 300}
]
}
}
}
\ No newline at end of file
# SynForest Configuration
This document provides details and instructions for the .json configuration files used in the forest generation process.
## General
The configuration file consists of three main sections: Map, Scene, and Simulation
![Process](./img/process.png)
### 1. Map
This part configures the forest map, i.e., the coordinates of each tree and their various properties such as species, height, and diameter at breast height. The workflow can directly use the output from `ForestFactory` which is a `.res` file.
Here is a table describing each parameter in the json config file under `map`:
| Parameter | Description |
| --------- | ----------- |
| `stand` | Path to the `.res` stand file that contains the information about the trees in the forest |
| `side` | Length of the side of the square map in meters |
| `species` | Identifier of the species that is being simulated, e.g. `QuePet` |
### 2. Scene
| Parameter | Description |
| --------- | ----------- |
| `pytreedb_dir` | Path to the PyTreeDB directory |
| `model_source` | Can be either `ULS`, `TLS` or `ALS` |
| `tile_w` | Width (x-length) of each tile in meters |
| `tile_h` | Height (y-length) of each tile in meters |
| `tile_overlap` | The overlap between adjacent tiles in meters |
| `template` | Contains parameters for the generated scene file for Helios++ |
| `template.id` | Scene ID |
| `template.name` | Scene name, can be same as ID |
| `template.area.z_var` | Variation of trees in the Z-axis |
| `template.trees` | Reserved |
### 3. Simulation
This part configures the simulation of the scene in Helios++ and path to the point cloud output.
| Parameter | Description |
| --------- | ----------- |
| `container` | Path to the container that contains the software for running the simulation |
| `lastools` | Path to the LASTools directory |
| `output_dir` | Path to directory to store simulation results |
| `template` | Contains parameters for simulation in Helios++ |
| `template.id` | ID of the survey |
| `template.name` | Name of the survey |
| `template.helios_seed` | Randomization seed for helios' noise and other effects |
| `template.platform.id` | Identifier of the platform (e.g. drone, plane) used for the survey |
| `template.platform.speed` | Speed of the platform in meters per second |
| `template.platform.altitude` | Altitude above ground of the platform in meters |
| `template.scanner.id` | Identifier of the scanner (e.g. lidar, camera) used for the survey |
| `survey.scanner.angle` | Angle from horizon of the scanner in degrees |
| `survey.scanner.pulse_freq` | Pulse frequency of the scanner in hertz |
| `survey.legs` | List of legs in the survey, each with its own coordinates in meters relative to scene origin |
docs/img/process.png

174 KiB

docs/img/synthetic_forest.png

832 KiB

docs/img/synthetic_forest_thick.png

3.19 MiB

absl-py==1.4.0
addict==2.4.0
anyio==3.6.1
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
asttokens==2.0.5
attrs==21.4.0
Babel==2.10.2
backcall==0.2.0
beautifulsoup4==4.11.1
bleach==5.0.0
cachetools==5.2.1
certifi==2022.5.18.1
cffi==1.15.0
charset-normalizer==2.0.12
cycler==0.11.0
debugpy==1.6.0
decorator==5.1.1
defusedxml==0.7.1
deprecation==2.1.0
docopt==0.6.2
entrypoints==0.4
executing==0.8.3
fastjsonschema==2.15.3
fonttools==4.33.3
google-auth==2.16.0
google-auth-oauthlib==0.4.6
grpcio==1.51.1
idna==3.3
importlib-metadata==4.11.4
importlib-resources==5.7.1
iniconfig==1.1.1
jedi==0.18.1
Jinja2==3.1.2
joblib==1.1.0
json5==0.9.8
jsonschema==4.6.0
kiwisolver==1.4.3
laspy==2.2.0
laszip==0.1.0
lazrs==0.4.5
Markdown==3.4.1
MarkupSafe==2.1.1
matplotlib==3.5.2
matplotlib-inline==0.1.3
mistune==0.8.4
nbclassic==0.3.7
nbclient==0.6.4
nbconvert==6.5.0
nbformat==5.4.0
nest-asyncio==1.5.5
notebook==6.4.12
notebook-shim==0.1.0
numpy==1.22.4
oauthlib==3.2.2
open3d==0.13.0
packaging==21.3
pandas==1.4.2
pandocfilters==1.5.0
parso==0.8.3
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.1.1
pluggy==1.0.0
prometheus-client==0.14.1
prompt-toolkit==3.0.29
protobuf==3.20.3
psutil==5.9.1
ptyprocess==0.7.0
pure-eval==0.2.2
py==1.11.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.21
Pygments==2.12.0
pyparsing==3.0.9
pyrsistent==0.18.1
pytest==7.1.2
python-dateutil==2.8.2
pytz==2022.1
PyYAML==6.0
pyzmq==23.1.0
requests==2.28.0
requests-oauthlib==1.3.1
rsa==4.9
scikit-learn==1.1.1
scipy==1.8.1
Send2Trash==1.8.0
six==1.16.0
sniffio==1.2.0
soupsieve==2.3.2.post1
stack-data==0.2.0
tensorboard==2.11.2
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
terminado==0.15.0
threadpoolctl==3.1.0
tinycss2==1.1.1
tomli==2.0.1
tomlkit==0.11.0
tornado==6.1
tqdm==4.64.0
traitlets==5.2.2.post1
urllib3==1.26.9
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==1.3.2
widgetsnbextension==3.6.0
yarg==0.1.9
zipp==3.8.0
import sys
import pandas as pd
import numpy as np
import os
import json
from copy import deepcopy
import argparse
import datetime
import itertools
import synforest.utils as utils
import synforest.steps as steps
if __name__ == '__main__':
if len(sys.argv) < 2:
print('Usage: python build_forest.py <config_path>')
sys.exit(1)
config = utils.load_config(sys.argv[1])
steps.map(config)
steps.scene(config)
steps.simulation(config)
steps.finalize(config)
This diff is collapsed.
############################################################
# Utils #
############################################################
# Get scripts root directory
root=$(dirname $(realpath $0 ))
############################################################
# Help #
############################################################
Help()
{
# Display Help
echo "Builds multiple forest tiles in parallel using helios container."
echo "---------"
echo "helios_multi_tiles.sh path_to_container work_dir assets_dir out_dir indexing lastools"
echo "---------"
echo " path_to_container Path to helios container"
echo " work_dir Path to project directory"
echo " assets_dir Path to assets directory"
echo " out_dir Path to output directory"
echo " indexing Tile indexes to build"
echo " lastools Path to lastools directory"
exit
}
Invalid()
{
echo "ERROR: Invalid arguments."
Help
}
Insuf()
{
echo "ERROR: Insufficient arguments."
Help
}
if [ $# -ge 6 ]; then
if [ $# -ge 7 ]; then
Invalid
fi
if [ $1 == 'help' ] || [ $1 == "-h" ]; then
Help
fi
else
Insuf
fi
CONTAINER_PATH="$1"
shift
CUR_WORK_DIR="$1"
FOREST_BASENAME=$(basename $CUR_WORK_DIR)
shift
ASSETS_DIR="$1"
shift
OUT_DIR="$1"
SCENE_FILENAME=/forestcare/scene/$FOREST_BASENAME
SIMULATION_FILENAME=/forestcare/simulation/$FOREST_BASENAME
shift
INDEXING="$1"
shift
LASTOOLS="$1"
echo "sbatch --parsable -a --mem-per-cpu=64G $INDEXING ./scripts/utils/sb_single_tile.sh $CONTAINER_PATH $CUR_WORK_DIR $ASSETS_DIR $SCENE_FILENAME $SIMULATION_FILENAME"
RES=$(sbatch --parsable -a "$INDEXING" ./scripts/utils/sb_single_tile.sh "$CONTAINER_PATH" "$CUR_WORK_DIR" "$ASSETS_DIR" "$SCENE_FILENAME" "$SIMULATION_FILENAME")
echo "Submitted job $RES"
echo "Waiting for job $RES to finish..."
sbatch --dependency=afterok:$RES "./scripts/utils/sb_merge_tiles.sh" "$CUR_WORK_DIR" "$OUT_DIR" "$LASTOOLS"
#sbatch --mem-per-cpu=64G "$FCDIR_CODE/forest_synthesis/scripts/utils/sb_merge_tiles.sh" "$CUR_WORK_DIR" "$OUT_DIR" "$LASTOOLS"
echo "Done."
\ No newline at end of file
#!/bin/bash
#SBATCH -t 8:00:00
#SBATCH -p standard96
#SBATCH -o tmp/%J-merge.txt
CUR_WORK_DIR="$1"
shift
OUT_DIR="$1"
shift
echo "Activating conda..."
CONDA_BASE=$(conda info --base)
source $CONDA_BASE/etc/profile.d/conda.sh
conda activate synforest
echo "Running merge_tiles.py"
python ./src/synforest/utils/merge_tiles.py $CUR_WORK_DIR $OUT_DIR $@
\ No newline at end of file
#!/bin/bash
#SBATCH -t 4:00:00
#SBATCH -p standard96
#SBATCH -o tmp/%J-tile.txt
CONTAINER_PATH="$1"
shift
CUR_WORK_DIR="$1"
shift
ASSETS_DIR="$1"
shift
SCENE_FILENAME="$1"
shift
SIMULATION_FILENAME="$1"
shift
TASK_ID="$SLURM_ARRAY_TASK_ID"
echo "Task ID: $TASK_ID"
echo "Loading singularity"
module load singularity
echo "singularity exec -B $HOME:/usr/users/$USER,$CUR_WORK_DIR:/forestcare,./src/synforest/utils/run_helios.py:/forestcare/run_helios.py,$ASSETS_DIR:/forestcare/assets $CONTAINER_PATH python /forestcare/run_helios.py $SCENE_FILENAME\_$TASK_ID.json $SIMULATION_FILENAME\_$TASK_ID.json"
singularity exec -B $HOME:/usr/users/$USER,$CUR_WORK_DIR:/forestcare,./src/synforest/utils/run_helios.py:/forestcare/run_helios.py,$ASSETS_DIR:/forestcare/assets,./template:/forestcare/template $CONTAINER_PATH python /forestcare/run_helios.py $SCENE_FILENAME\_$TASK_ID.json $SIMULATION_FILENAME\_$TASK_ID.json
echo -ne "Done @"
hostname
echo ""
\ No newline at end of file
import setuptools
setuptools.setup(
name="synforest",
version="0.2",
author="Ali Doosthosseini",
author_email="doosth.ali@gmail.com",
package_dir={'':'src'},
packages=["synforest", 'synforest.steps', 'synforest.utils'],
description="Synthetic Forest Generation",
long_description="Synthetic Forest Generation",
long_description_content_type="text/markdown",
license='MIT',
python_requires='>=3.8',
install_requires=[]
)
from . import steps
\ No newline at end of file
from .map import map
from .scene import scene
from .simulation import simulation
from .finalize import finalize
\ No newline at end of file