Gitlab Community Edition Instance

Commit a4fc6fb1 authored by mhellka's avatar mhellka
Browse files

Merge branch 'everything-changed' into 'master'

Massive rewrite of the CLI and lots of refactoring work.

See merge request !8
parents 86d54c54 de66c07a
......@@ -11,4 +11,5 @@ MANIFEST
......@@ -7,7 +7,8 @@ venv/touch-me: requirements.txt
venv/bin/pip install -U pip -r requirements.txt
touch venv/touch-me
docs: venv $(DOCS_ALL) $(SRC_ALL)
docs: build/docs/index.html
build/docs/index.html: venv $(DOCS_ALL) $(SRC_ALL)
venv/bin/sphinx-build -a docs/ build/docs/
dist: venv $(SRC_ALL)
......@@ -5,38 +5,69 @@ This is a client library and command-line toolbelt for accessing a
There is already a library called [pycdstar]( for accessing older versions of CDSTAR. The two libraries may be merged at some point, but for now, use [pycdstar]( for CDSTAR 2 and [pycdstar3]( for CDSTAR 3.
## Command-Line interface
## Library
`pycdstar3` is also a command-line toolbox to upload, download or manage data in a CDSTAR repository.
The `pycdstar3.CDStar` class closely reassembles the actual CDSTAR v3 REST API, one method per API endpoint. It provides basic connection pooling, transaction management, error handling and json parsing on top of the `requests` library, but tries to stay out of your way otherwise. Use this if you already know the CDSTAR API and need maximum control.
from pycdstar3 import CDStar
cdstar = CDStar("", auth=("USER", "PASS"))
with cdstar.begin(autocommit=True): # Wrap everything in a transaction (optional)
vault_name = "demo"
archive_id = cdstar.create_archive(vault_name).id
cdstar.put_file(vault_name, archive_id, "", __file__)
with cdstar.get_file(vault_name, archive_id, "") as download:
with open("/tmp/", "wb") as target:
for chunk in download.iter_content(buffsize=1024*64):
Please note that the command-line interface is made for humans, not scripts. The commands and output may change between releases without notice. If you want to automate CDSTAR, consider implementing your tools directly against the `pycdstar3` client library. Libraries for other languages may also be available. As a last resort, you can always develop directly against the stable [CDSTAR REST API]( using the HTTP client of your choice.
For a more fluent and high-level approach, wrap your `CDStar` instance in a `CDStarVault`. This hides the HTTP layer behind an object-oriented API and offers convenient wrappers and methods for the most common operations.
### cdstar.conf
from pycdstar3 import CDStar, CDStarVault
cdstar = CDStar("", auth=("USER", "PASS"))
The `pycdstar3` client will look for a `cdstar.conf` in the current directory or its parent directories and try to load default values from it. The most important settings are the CDSTAR server URI and a default vault name. If these are defined, you can reference archives by ID only, instead of the full service URI. This saves a *lot* of typing. The `pycdstar init` command will help you create this file.
with cdstar.begin(autocommit=True): # Wrap everything in a transaction (optional)
vault = CDStarVault(cdstar, "demo")
archive = vault.create_archive()
archive.put("", __file__)
### Referencing Vaults, Archives and Files
with archive.file("").download() as download:
Most `pycdstar3` client commands operate on a specific vault, archive or file. These can always be referenced by their full URI. If a `cdstar.conf` is present and default values for server and vault are defined, then also a couple of short forms can be used. The following syntax forms are supported:
## Command-Line interface
`pycdstar3` is also a command-line toolbox to upload, download or manage data in a CDSTAR repository.
* **Full URI**: If the reference starts with `http(s)://` then everything up to the first `/v3/` is used as the server URI, followed by a vault and optionally an archive ID and file path. The default server and vault settings are not used.
Please note that the command-line interface is made for humans, not scripts. The commands and output may change between releases. If you want to automate CDSTAR, consider implementing your tools directly against the `pycdstar3` client library.
Example: `pycdstar3 get`
* **Vault Name**: If the reference starts with a forward slash and a vault name, then the default server from `cdstar.conf` is used, but the default vault is ignored. To reference the default vault, leave the vault name empty.
### Workspace mode
Example: `pycdstar3 get /myvault/e497a76f/some/file.txt`
Example: `pycdstar3 get //e497a76f/some/file.txt` (default vault)
* **Archive ID**: In the shortest form, the reference must start with an archive ID and both default server and vault settings must be present in your `cdstar.conf`.
Most `pycdstar3` commands require `--server` and `--vault` arguments to be set, or `CDSTAR_SERVER` and `CDSTAR_VAULT` environment variables to be defined. As an alternative, you can run `pycdstar3 init` to create a *workspace directory*. When within this directory (or any sub-directory), the client will switch to *workspace mode* and read default values for `--server`, `--vault` and other settings from workspace configuration. This is particularly useful if you are working with multiple servers or vaults and want to easily switch between them.
Example: `pycdstar3 get e497a76f/some/file.txt`
# Explicit
pycdstar3 --server --vault myVault search 'dcAuthor=Einstein'
Some commands will accept a special string `new` as an archive ID and create a new archive on the fly.
# Environment Variables
export CDSTAR_VAULT=myVault
pycdstar3 search 'dcAuthor=Einstein'
# Workspace mode
pycdstar3 init # creates .cdstar/pycdstar.conf
pycdstar3 search 'dcAuthor=Einstein'
### Command Usage
This is an (incomplete) list of (planned) commands. For a complete list, run `pycdstar3 -h` and for details, see `pycdstar3 COMMAND -h`.
* **`init`**: Ask for server address, vault, credentials and other config options and create a `cdstar.conf` file in the current directory.
* **`init`**: Create a new workspace.
* **`new`** Create a new (empty) archive.
* **`info`** Query information about vaults, archives or files.
* **`meta get/set/list`** Manage metadata attributes for archives or files.
......@@ -50,7 +81,6 @@ This is an (incomplete) list of (planned) commands. For a complete list, run `py
* **`scroll`** List all IDs in a vault.
* **`search`** Search in a vault.
## License
Copyright 2019 Gesellschaft für wissenschaftliche Datenverarbeitung mbH Göttingen
API reference
.. automodule:: pycdstar3
API Client (low-level)
.. autoclass:: pycdstar3.api.CDStar
Resource Handles (fluent API)
.. autoclass:: pycdstar3.api.CDStarVault
.. autoclass:: pycdstar3.api.CDStarArchive
.. autoclass:: pycdstar3.api.CDStarFile
Data Classes and Wrappers
.. automodule:: pycdstar3.model
......@@ -36,6 +36,12 @@ extensions = [
autodoc_default_options = {
'members': True,
'member-order': 'bysource',
'undoc-members': True
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
......@@ -5,7 +5,7 @@ Welcome to pycdstar3's documentation!
:maxdepth: 2
:caption: Contents:
Indices and tables
.. automodule:: pycdstar3.client
API reference
.. toctree::
:caption: Modules:
__version__ = "3.0.dev0"
__url__ = ""
from pycdstar3.client import * # noqa: F401, F403
from pycdstar3.api import * # noqa: F403 F401
import pycdstar3.api
__all__ = pycdstar3.api.__all__
import os
import urllib.parse
PATH_TYPES = (str,)
if hasattr(os, "PathLike"):
PATH_TYPES = (str, os.PathLike) # pragma: no cover
class cached_property(object):
def __init__(self, func):
self.__doc__ = getattr(func, '__doc__')
self.func = func
def __get__(self, obj, cls):
if obj is None:
return self
value = obj.__dict__[self.func.__name__] = self.func(obj)
return value
def url_split_auth(url: str):
""" Extract and remove auth information from an URL. Return a (cleaned-url, username, password) tuple.
The cleaned-url will have any auth information removed from the netloc part.
Username and password may be None if not present.
split = urllib.parse.urlsplit(url)
username, password = split.username, split.password
if username or password:
netloc =
if split.port:
netloc += ":" + str(split.port)
url = urllib.parse.urlunsplit(split._replace(netloc=netloc))
return url, username, password
......@@ -4,20 +4,17 @@ CDSTAR command-line client
import argparse
import importlib
import json
import os
import sys
from pycdstar3 import __version__ as VERSION
from pycdstar3.client import ApiError
from pycdstar3.cli._utils import Printer
from pycdstar3 import __version__ as VERSION, ApiError
__ALL__ = ["main", "printer"]
parser = argparse.ArgumentParser(prog="pycdstar3")
parser.add_argument("-C", metavar="DIR", default=".",
help="Change the current working directory before executing the command")
parser.add_argument("-c", "--config", metavar="FILE",
help="Path to config file (default: find automatically)")
parser.add_argument("--server", metavar="URI",
help="CDSTAR server URI (e.g. Required for most commands.")
parser.add_argument("--vault", metavar="NAME",
help="Vault to work with. required for most commands.")
parser.add_argument("--version", action="store_true", help="Print version and exit")
_grp = parser.add_mutually_exclusive_group()
_grp.add_argument("-v", "--verbose", action="count", default=0,
......@@ -27,10 +24,6 @@ subparsers = parser.add_subparsers(title="available commands",
description='Run "COMAMND -h" to get help for a specific command.',
#: This should be used to print optional messages to the user.
#: Messages are printed to stderr, so only use it for complementary information, not for the primary results.
printer = Printer(level=0, file=sys.stderr)
def _autodiscover_commands():
""" Autodiscover and import all modules in the pycdstar3.cli.commands namespace.
......@@ -59,39 +52,29 @@ def main(*args): # noqa: C901
print("pycdstar3-" + VERSION)
# Set root logging level based on verbosity setting
if opts.quiet:
if opts.C != ".":
printer.v("Changing working directory to: {}", opts.C)
if not hasattr(opts, "main"):
# No sub-command specified
return 1
from pycdstar3.cli.context import CliContext
ctx = CliContext(opts, workdir=".")
from pycdstar3.cli.context import CliContext
ctx = CliContext(workdir=".")
if opts.config:
return opts.main(ctx, opts) or 0
except KeyboardInterrupt:
return 0
except CliError as e:
return e.return_code
except ApiError as e:
printer.v("Full error response:")
printer.v(json.dumps(e.json, indent=2), indent=2)
ctx.print.v("Full error response:")
ctx.print.v(json.dumps(e.json, indent=2), indent=2)
return e.status
except Exception as e:
printer.fatal("Uncaught exception ({}: {}). Exiting...", type(e).__name__, e)
ctx.print.fatal("Uncaught exception ({}: {}). Exiting...", type(e).__name__, e)
return 1
......@@ -88,7 +88,7 @@ class Printer:
set_verbosity = __init__
def _print(self, msg, *args, **kwargs):
def _print(self, msg="", *args, **kwargs):
hr = kwargs.pop("highlight", None)
indent = kwargs.pop("indent", 0)
if args:
......@@ -104,22 +104,22 @@ class Printer:
import shutil
return shutil.get_terminal_size((40, 20))[0]
def __call__(self, msg, *args, **kwargs):
def __call__(self, msg="", *args, **kwargs):
""" Print only if -q (--quiet) was NOT passed as a command-line parameter """
if self.verbosity >= 0:
self._print(msg, *args, **kwargs)
def v(self, msg, *args, **kwargs):
def v(self, msg="", *args, **kwargs):
""" Print only if -v was passed as a command-line parameter """
if self.verbosity >= 1:
self._print(msg, *args, **kwargs)
def vv(self, msg, *args, **kwargs):
def vv(self, msg="", *args, **kwargs):
""" Print only if -vv was passed as a command-line parameter """
if self.verbosity >= 2:
self._print(msg, *args, **kwargs)
def vvv(self, msg, *args, **kwargs):
def vvv(self, msg="", *args, **kwargs):
""" Print only if -vvv was passed as a command-line parameter """
if self.verbosity >= 3:
self._print(msg, *args, **kwargs)
Manage access control lists (ACL)
from pycdstar3.client import FormUpdate
from pycdstar3.cli import printer
from pycdstar3 import FormUpdate
def register(subparsers):
......@@ -10,21 +9,23 @@ def register(subparsers):
sub = parser.add_subparsers()
pset = sub.add_parser("set")
pset.add_argument("TARGET", help="Archive ID")
pset.add_argument("ARCHIVE", help="Archive ID")
pset.add_argument("SUBJECT", help="Subject do modify")
pset.add_argument("PERMISSIONS", nargs="*", help="Permissions to set for this subject.")
def acl_set(ctx, args):
client, vault, archive, rfile = ctx.resolve(args.TARGET)
client = ctx.client
vault = ctx.vault
archive = args.ARCHIVE
subject = args.SUBJECT
permissions = args.PERMISSIONS
printer("Set ACL for {!r} on /{}/{} -> [{}]", subject, vault, archive, ', '.join(permissions))
with client.begin(autocommit=True):
ctx.print("Set ACL for {!r} on /{}/{} -> [{}]", subject, vault, archive, ', '.join(permissions))
with archive.api.begin(autocommit=True):
update = FormUpdate()
update.acl(subject, *permissions)
client.update_archive(vault, archive, form=update)
......@@ -5,7 +5,7 @@ import os
import sys
from pycdstar3.cli._utils import hbytes
from pycdstar3.cli import printer, CliError
from pycdstar3.cli import CliError
def register(subparsers):
......@@ -17,13 +17,18 @@ def register(subparsers):
" interrupted download. Also, keep the *.part file"
" on any errors.")
parser.add_argument("-f", "--force", action="store_true", help="Overwrite local files without asking")
parser.add_argument("SRC", help="File identifier")
parser.add_argument("ARCHIVE", help="Archive ID")
parser.add_argument("FILE", help="File Name")
parser.add_argument("DST", nargs="?", help="Destination filename, directory or '-' for stdout. (default: '-')")
def get(ctx, args): # noqa: C901
client, vault, archive, file = ctx.resolve(args.SRC)
client = ctx.client
vault = ctx.vault
archive = args.ARCHIVE
file = args.FILE
resume = args.resume
progress = args.progress
dst = args.DST or '-'
......@@ -45,7 +50,7 @@ def get(ctx, args): # noqa: C901
if partfile and os.path.exists(partfile):
if resume:
offset = os.path.getsize(partfile)
printer("Resuming download at: {}", hbytes(offset))
ctx.print("Resuming download at: {}", hbytes(offset))
raise CliError("Found partial download, but --resume is not set: " + partfile)
......@@ -62,10 +67,10 @@ def get(ctx, args): # noqa: C901
if ispipe and out.isatty() and not dl.type.startswith("text/"):
raise CliError("Not printing binary data ({}) a terminal".format(dl.type))
if progress and not printer.quiet:
if progress and not ctx.print.quiet:
from tqdm import tqdm
pbar = tqdm(total=dl.size+offset, initial=offset, unit='b', unit_scale=True, unit_divisor=1024,
dynamic_ncols=True, file=printer.file)
dynamic_ncols=True, file=ctx.print.file)
def write(chunk):
......@@ -75,7 +80,7 @@ def get(ctx, args): # noqa: C901
printer.v("Downloading {} ({})", file, hbytes(dl.size + offset))
ctx.print.v("Downloading {} ({})", file, hbytes(dl.size + offset))
for chunk in dl.iter_content():
......@@ -87,7 +92,7 @@ def get(ctx, args): # noqa: C901
os.rename(partfile, dst)
printer.v("Done!", file, vault, archive, hbytes(dl.size + offset))
ctx.print.v("Done!", file, vault, archive, hbytes(dl.size + offset))
except (KeyboardInterrupt, Exception):
if close:
......@@ -96,6 +101,6 @@ def get(ctx, args): # noqa: C901
except OSError:
printer("Failed to delete partial download: {}", partfile)
ctx.print("Failed to delete partial download: {}", partfile)
......@@ -8,59 +8,68 @@ If the main --config parameter is set, the configuration is saved at the specifi
instead of the current working directory.
import configparser
import os
import re
from pycdstar3.cli import printer
from pycdstar3.cli.context import CONFIG_NAMES
from pycdstar3.api import CDStar
from pycdstar3.cli import CliError
from pycdstar3.cli.context import Workspace
def register(subparsers):
parser = subparsers.add_parser("init", help=__doc__.strip().splitlines()[0], description=__doc__)
parser.add_argument("--server", help="CDSTAR server URI (usually ends in '/v3')")
parser.add_argument("--vault", help="Name of the default vault")
grp = parser.add_mutually_exclusive_group()
grp.add_argument("--token", help="Auth token to use")
grp.add_argument("--auth", help="Login credentials as a username:password string")
parser.set_defaults(main=command, noconfig=True)
parser.add_argument("PATH", nargs="?", help="Folder to turn into a workspace directory (default: '.')", default=".")
def command(ctx, args):
root = os.path.abspath(args.PATH)
work = Workspace(root)
target = os.path.abspath(args.config or CONFIG_NAMES[0])
if os.path.exists(target):
raise ValueError("Config file '{}' already exists. Modify it with an editor.".format(target))
server = args.server or ask("Server URI", "", r"^https?://.*/v3$")
vault = args.vault or ask("Default vault name", "MyVault", r"^.+$")
if args.token:
auth = ("token", args.token)
elif args.auth:
auth = ("basic", args.auth)
method = ask("Auth method (basic or token)", "basic", r"^(basic|token)$")
if method == "basic":
auth = (method, ask("Username") + ":" + ask("Password"))
auth = (method, ask("Token"))
if os.path.exists(work.configdir) or os.path.exists(work.configfile):
raise CliError("Config directory '{}' already exists.".format(work.configdir))
ctx.print("Creating workspace in: {}", work.root)
server = ask("CDSTAR Server URI", args.server, r"^https?://.*/?$")
if not server.endswith("/"):
server += "/"
if not server.endswith("v3/"):
server += "v3/"
ctx.print(" Connecting to {} ... ", server, end="")
info = CDStar(server).service_info()
ctx.print("OK ({})", info.version.cdstar)
vaults = info.vaults or []
ctx.print(" Available vaults: {}", ', '.join(sorted(vaults)))
except Exception:
raise CliError("Failed to connect")
config = configparser.ConfigParser()
config["DEFAULT"]["server"] = server
config["DEFAULT"]["vault"] = vault
if auth:
config["DEFAULT"]["auth"] = ':'.join(auth)
vault = ask("Select a vault", args.vault or (info.vaults and info.vaults[0]), choice=vaults, rx=".+")
with open(target, 'x') as fp:
fp.write("# cdstar-cli config. See\n")
work.store_config(server=server, vault=vault)
printer("Config file written to: {}", target)
ctx.print.v("Config saved to: {}", work.configfile)
ctx.print("Created workspace directory: {}", work.root)
def ask(q, default=None, rx=None):
def ask(q, default=None, rx=None, choice=None, password=False):
import readline # noqa: F401
except ImportError:
while True:
val = input("{}? [{}] ".format(q, default) if default else "{}? ".format(q)).strip()
prompt = "{} [{}]: ".format(q, default) if default else "{}: ".format(q)
if password:
import getpass
val = getpass.getpass(prompt)
val = input(prompt)
if not val:
if default:
return default
......@@ -69,4 +78,8 @@ def ask(q, default=None, rx=None):
if rx and not re.match(rx, val):
print("This does not look right. Try again...")
if choice and val not in choice:
print("Please select one of: {}", choice)
return val
......@@ -4,15 +4,13 @@ List files in an archive or in a sub directory.
from collections import defaultdict
import iso8601 as iso8601
import iso8601
from pycdstar3.cli import printer
from pycdstar3.cli._utils import hbytes
def register(subparsers):
parser = subparsers.add_parser("ls", help=__doc__.strip().splitlines()[0], description=__doc__)
parser.add_argument("REF", help="Archive to list")
parser.add_argument("-f", "--format", default="name",
help="Change what is printed per file. provide either a python format string, "
"or a csv with field names. Available fields: "
......@@ -26,11 +24,17 @@ def register(subparsers):
help="Include files by glob pattern (default: all)")
parser.add_argument("-x", "--exclude", metavar="GLOB", action="append",
help="Exclude files by glob pattern")
parser.add_argument("ARCHIVE", help="Archive to list")
parser.add_argument("PREFIX", nargs="?", help="Directory to list", default="/")
def ls(ctx, args):
client, vault, archive, prefix = ctx.resolve(args.REF)
client = ctx.client
vault = ctx.vault