Gitlab Community Edition Instance

Commit 5ad25cea authored by Marcel Hellkamp's avatar Marcel Hellkamp
Browse files

Prototyping

parent 88273d3e
...@@ -15,19 +15,30 @@ and newer. ...@@ -15,19 +15,30 @@ and newer.
`cdstar3` is a command-line toolbox to upload, download or manage data in a CDSTAR repository. `cdstar3` is a command-line toolbox to upload, download or manage data in a CDSTAR repository.
Please note that `cdstar3` was designed to be used by humans, not scripts. The output is mostly human-friendly and may Please note that `cdstar3` was designed to be used by humans, not scripts. The output is mostly human-friendly and may change between releases. If you want to automate CDSTAR, consider implementing your tools using the `pycdstar3` client library, or directly against the stable CDSTAR REST API.
change between releases. If you want to automate CDSTAR, consider implementing your tools using the `pycdstar3` client library, or directly against the stable CDSTAR REST API.
### Configuration ### Configuration
The `cdstar3` client needs to know which server to connect to and which vault to use per default. This information (and more) is defined in a file named `cdstar.conf`. If no such file is specified via the `-c` parameter, `cdstar3` will look for a `cdstar.conf` in the current working directory, and then in any of its parent directories. As a last resort, certain system-dependent user folders are searched (e.g. `~/.config/cdstar3/` on linux). You can create a configuration file with `cdstar3 init`. The `cdstar3` client needs to know which server to connect to and which vault to use per default. This information (and more) is defined in a file named `cdstar.conf`. If no such file is specified via the `-c` parameter, `cdstar3` will look for a `cdstar.conf` in the current working directory, and then in any of its parent directories. As a last resort, certain system-dependent user folders are searched (e.g. `~/.config/cdstar3/` on linux). You can create a configuration file with `cdstar3 init`.
### Target Strings
A single CDSTAR server might host multiple vaults, each containing any number of archives, each containing multiple files. These can be referenced using their full resource URI (e.g. `http(s)://$server/v3/$vault/$archive/$file`) but that would be a lot of typing if you are mostly working with a single vault at a time. For this reason, a default server and vault can be configured in your `cdstar.conf` and the target strings will get a lot shorter: Archives can be references by their ID alone, and files by `$archive/$file`. If you want to specify a different vault, you can use the slight longer `/$vault/$archive` or `/$vault/$archive/$file` forms. Notice the leading `/` if you specify a vault. Even with a `cdstar.conf` present, you can still use the full URI if needed.
### Usage ### Usage
This is an (incomplete) list of commands and their most important parameters. For a complete list, run `cdstar3 -h` and for details, see `cdstar3 COMMAND -h`. This is an (incomplete) list of commands and their most important parameters. For a complete list, run `cdstar3 -h` and for details, see `cdstar3 COMMAND -h`.
* **`init`**: Ask for server address, vault, credentials and other config options and create a `cdstar.conf` file in the current directory. * **`init`**: Ask for server address, vault, credentials and other config options and create a `cdstar.conf` file in the current directory.
* **`put ID [path]`** Upload one or more files or folders to an archive.
* **`get ID/FILE (path)`** Download a single file. If no path is specified, it is streamed to stdout.
* **`info ID[/FILE]`** Get information about an archive or file.
* **`meta get ID[/FILE] (FIELD)`** Get metadata about an archive or file.
* **`meta set ID[/FILE] [FIELD=VALUE]`** SET metadata about an archive or file.
###########################################
High-level commands work with local archive directories (one per archive). When creating a new archive or recovering an existing archive for the first time, `cdstar3` will create a hidden `.cdstar` folder within the target directory and remember the exact location (server, vault and id) of the remote archive. Do NOT delete this folder, or the correlation between your local copy and the remote archive is lost. High-level commands work with local archive directories (one per archive). When creating a new archive or recovering an existing archive for the first time, `cdstar3` will create a hidden `.cdstar` folder within the target directory and remember the exact location (server, vault and id) of the remote archive. Do NOT delete this folder, or the correlation between your local copy and the remote archive is lost.
* **`archive DIR`**: Create a new remote archive from a local directory. * **`archive DIR`**: Create a new remote archive from a local directory.
......
...@@ -17,6 +17,7 @@ class CDStar: ...@@ -17,6 +17,7 @@ class CDStar:
def __init__(self, url, auth=None, _session=None): def __init__(self, url, auth=None, _session=None):
self.url = url.rstrip("/") + '/' self.url = url.rstrip("/") + '/'
self.auth = auth self.auth = auth
self.headers = {}
self._session = _session or requests.Session() self._session = _session or requests.Session()
self._tx = None self._tx = None
...@@ -32,8 +33,8 @@ class CDStar: ...@@ -32,8 +33,8 @@ class CDStar:
if self.auth: if self.auth:
options['auth'] = self.auth options['auth'] = self.auth
if self._tx: if self.headers:
options.setdefault("headers", {})["X-Transaction"] = self._tx['id'] options.setdefault("headers", {}).update(self.headers)
rs = self._session.request( rs = self._session.request(
method, self.url + '/'.join(path), **options) method, self.url + '/'.join(path), **options)
...@@ -43,38 +44,15 @@ class CDStar: ...@@ -43,38 +44,15 @@ class CDStar:
# TODO: handle errors # TODO: handle errors
raise Exception(rs.text) raise Exception(rs.text)
def begin(self, autocommit=False): @property
""" Begin a new transaction, return self """ def tx(self):
if self._tx: return self._tx
self.rollback()
self._tx = self._rest("POST", "_tx").json()
self._tx['x-autocommit'] = autocommit
return self
def commit(self):
if not self._tx:
raise RuntimeError("No transaction running")
self._rest("POST", "_tx", self._tx['id'])
self._tx = None
def rollback(self):
""" Rollback the current transaction.
Do nothing if no transaction is active.
"""
if self._tx:
self._rest("DELETE", "_tx", self._tx['id'])
self._tx = None
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, traceback): def begin(self, autocommit=False, readonly=True):
if self._tx: if self._tx:
if exc_type is None and self._tx['x-autocommit']: self._tx.rollback()
self.commit() self._tx = TxHandle(self, autocommit=autocommit, readonly=readonly)
else: return self._tx
self.rollback()
def service_info(self): def service_info(self):
return self._rest("GET").json() return self._rest("GET").json()
...@@ -100,6 +78,59 @@ class CDStar: ...@@ -100,6 +78,59 @@ class CDStar:
) )
class TxHandle:
def __init__(self, client, autocommit=False, readonly=True):
self.client = client
self.autocommit = autocommit
self.readonly=readonly
self._tx = None
def __enter__(self):
if self.client._tx:
self.client._tx.rollback()
self._tx = self.client._rest("POST", "_tx", data={"readonly": self.readonly}).json()
self._tx['x-autocommit'] = self.autocommit
self.client._tx = self
return self
def __exit__(self, exc_type, exc_value, traceback):
if self._tx and self.client._tx is self:
if exc_type is None and self._tx['x-autocommit']:
self.commit()
else:
self.rollback()
self.client._tx = None
@property
def id(self):
return self._tx['id']
@property
def is_running(self):
return bool(self._tx)
def renew(self):
if not self._tx:
raise RuntimeError("No transaction running")
self.client._rest("GET", "_tx", self._tx['id'])
def commit(self):
if not self._tx:
raise RuntimeError("No transaction running")
self.client._rest("POST", "_tx", self._tx['id'])
self._tx = None
def rollback(self):
""" Rollback the current transaction.
Do nothing if no transaction is active.
"""
if self._tx:
self.client._rest("DELETE", "_tx", self._tx['id'])
self._tx = None
class PostUpdate: class PostUpdate:
""" Builder for CDSTAR POST multipart/form-data requests to upload multiple files or change aspects of an archive. """ Builder for CDSTAR POST multipart/form-data requests to upload multiple files or change aspects of an archive.
""" """
......
...@@ -4,10 +4,6 @@ import importlib ...@@ -4,10 +4,6 @@ import importlib
import os import os
import re import re
import sys import sys
from typing import Tuple, Dict
from .context import CliContext
from ..cdstar import CDStar
__ALL__ = ["main", "register_subcommand", "printer"] __ALL__ = ["main", "register_subcommand", "printer"]
...@@ -89,7 +85,7 @@ class Printer: ...@@ -89,7 +85,7 @@ class Printer:
import traceback import traceback
traceback.print_exc(file=self.file) traceback.print_exc(file=self.file)
else: else:
self("Stacktrace not shown. Add -v to print a full stacktrace.") self("Stacktrace not shown. Add -vv to print a full stacktrace.")
#: This should be used to print optional messages to the user. #: This should be used to print optional messages to the user.
...@@ -112,7 +108,7 @@ def main(args=None): ...@@ -112,7 +108,7 @@ def main(args=None):
printer.set_verbosity(opts.verbose) printer.set_verbosity(opts.verbose)
if opts.C != ".": if opts.C != ".":
printer.v("Changing working directory to:", opts.C) printer.v("Changing working directory to: {}", opts.C)
os.chdir(opts.C) os.chdir(opts.C)
if not hasattr(opts, "subcommand"): if not hasattr(opts, "subcommand"):
...@@ -124,6 +120,8 @@ def main(args=None): ...@@ -124,6 +120,8 @@ def main(args=None):
try: try:
ctx = CliContext(workdir=".") ctx = CliContext(workdir=".")
if opts.config:
ctx.load_config(opts.config)
return cmdmain(ctx, opts) or 0 return cmdmain(ctx, opts) or 0
except KeyboardInterrupt: except KeyboardInterrupt:
printer("Exiting...") printer("Exiting...")
...@@ -132,13 +130,10 @@ def main(args=None): ...@@ -132,13 +130,10 @@ def main(args=None):
printer.fatal(*e.args) printer.fatal(*e.args)
return e.return_code return e.return_code
except Exception as e: except Exception as e:
printer.fatal("Uncaught exception. Exiting...") printer.fatal("Uncaught exception ({}). Exiting...", e)
return 1 return 1
class CliError(Exception): class CliError(Exception):
""" Exception that will cause a clean command-line shutdown without any stack trace. """ Exception that will cause a clean command-line shutdown without any stack trace.
...@@ -147,3 +142,5 @@ class CliError(Exception): ...@@ -147,3 +142,5 @@ class CliError(Exception):
def __init__(self, *args, status=1): def __init__(self, *args, status=1):
super().__init__(*args) super().__init__(*args)
self.return_code = status self.return_code = status
from .context import CliContext
...@@ -12,8 +12,8 @@ import configparser ...@@ -12,8 +12,8 @@ import configparser
import os import os
import re import re
from cli import CONFIG_NAMES, register_subcommand, printer from .. import register_subcommand, printer
from ..context import CONFIG_NAMES
def register(): def register():
parser = register_subcommand("init", command, help=__doc__.splitlines()[0], description=__doc__, noconfig=True) parser = register_subcommand("init", command, help=__doc__.splitlines()[0], description=__doc__, noconfig=True)
......
...@@ -3,36 +3,37 @@ Upload one or more files to an existing archive. ...@@ -3,36 +3,37 @@ Upload one or more files to an existing archive.
""" """
import os import os
from cli import register_subcommand, compile_glob, FileProgress, hbytes, printer, CliError from ..context import VerboseTx
from .. import register_subcommand, printer, CliError
from .._utils import compile_glob, hbytes
def register(): def register():
parser = register_subcommand("put", command, help=__doc__.splitlines()[0], description=__doc__) parser = register_subcommand("put", command, help=__doc__.splitlines()[0], description=__doc__)
parser.add_argument("-r", "--recursive", action="store_true", help="Upload entire directories")
parser.add_argument("-a", "--all", action="store_true", help="Include hidden files (skipped by default)") parser.add_argument("-a", "--all", action="store_true", help="Include hidden files (skipped by default)")
parser.add_argument("-n", "--dry-run", action="store_true", parser.add_argument("-n", "--dry-run", action="store_true",
help="Do not upload anything, just print what would have been uploaded") help="Do not upload anything, just print what would have been uploaded")
parser.add_argument("--flat", action="store_true", parser.add_argument("--flat", action="store_true",
help="Strip all directory names from the local path and store all files into the root path (e.g. ./path/to/file.txt would be uploaded as /file.txt)") help="Strip all directory names from the local path and store all files into the root path (e.g. ./path/to/file.txt would be uploaded as /file.txt)")
parser.add_argument("-x", "--exclude", metavar="GLOB", action="append", help="Exclude files by glob pattern") parser.add_argument("-x", "--exclude", metavar="GLOB", action="append",
help="Exclude files by glob pattern")
parser.add_argument("-i", "--include", metavar="GLOB", action="append", parser.add_argument("-i", "--include", metavar="GLOB", action="append",
help="Include files by glob pattern (default: all)") help="Include files by glob pattern (default: all)")
parser.add_argument("-p", "--progress", action="store_true", parser.add_argument("-p", "--progress", action="store_true",
help="Show progress bar for large files or slow uploads") help="Show progress bar for large files or slow uploads")
parser.add_argument("archive", help="Archive ID, or 'new' to create a new archive") parser.add_argument("TARGET", help="Archive ID, or 'new' to create a new archive")
parser.add_argument("file", nargs='+', help="File(s) (or directories) to upload") parser.add_argument("PATH", nargs='+', help="Files or directories to upload")
parser.set_defaults(cmd=command) parser.set_defaults(cmd=command)
def findfiles(flist, recursive=False, include_hidden=False): def findfiles(flist, include_hidden=False):
def keep(name): def keep(name):
return include_hidden or not name.startswith('.') return include_hidden or not name.startswith('.')
for entry in flist: for entry in flist:
if os.path.isfile(entry): if os.path.isfile(entry):
yield entry yield entry
elif os.path.isdir(entry) and recursive: elif os.path.isdir(entry):
for (root, dirs, files) in os.walk(entry): for (root, dirs, files) in os.walk(entry):
dirs[:] = filter(keep, dirs) dirs[:] = filter(keep, dirs)
files[:] = filter(keep, files) files[:] = filter(keep, files)
...@@ -43,15 +44,17 @@ def findfiles(flist, recursive=False, include_hidden=False): ...@@ -43,15 +44,17 @@ def findfiles(flist, recursive=False, include_hidden=False):
def command(ctx, args): def command(ctx, args):
archive = args.archive client, vault, archive, rfile = ctx.resolve(args.TARGET)
vault = ctx.default_vault
inc = [compile_glob(rule).match for rule in args.include or []] inc = [compile_glob(rule).match for rule in args.include or []]
exc = [compile_glob(rule).match for rule in args.exclude or []] exc = [compile_glob(rule).match for rule in args.exclude or []]
progress = args.progress progress = args.progress
if not rfile:
rfile = '/'
uploads = {} uploads = {}
total = 0 total = 0
files = findfiles(args.file, args.recursive, args.all) files = findfiles(args.PATH, args.all)
for file in files: for file in files:
if inc and not any(rule(file) for rule in inc): if inc and not any(rule(file) for rule in inc):
printer.vv("Skipping: {} (not included)", file) printer.vv("Skipping: {} (not included)", file)
...@@ -60,9 +63,16 @@ def command(ctx, args): ...@@ -60,9 +63,16 @@ def command(ctx, args):
printer.vv("Skipping: {} (excluded)", file) printer.vv("Skipping: {} (excluded)", file)
continue continue
target = os.path.join("/", os.path.relpath(file)) target = os.path.relpath(file)
if args.flat: if args.flat:
target = "/" + os.path.basename(target) target = os.path.basename(target)
if rfile.endswith("/"):
target = os.path.join(rfile, target)
else:
target = rfile
if target != os.path.normpath(target):
raise ValueError("Unable to create remote file with relative or empty path segments: {}".format(target))
if target in uploads and uploads[target][0] != file: if target in uploads and uploads[target][0] != file:
raise ValueError("File included twice: {} and {} both map to {}".format(file, uploads[target][0], target)) raise ValueError("File included twice: {} and {} both map to {}".format(file, uploads[target][0], target))
stat = os.stat(file) stat = os.stat(file)
...@@ -76,9 +86,9 @@ def command(ctx, args): ...@@ -76,9 +86,9 @@ def command(ctx, args):
len(uploads), hbytes(total), "new archive" if archive == 'new' else "archive: " + vault + "/" + archive) len(uploads), hbytes(total), "new archive" if archive == 'new' else "archive: " + vault + "/" + archive)
return return
with ctx.client.begin(autocommit=True) as ctx: with VerboseTx(client.begin(autocommit=True)) as tx:
if archive == 'new': if archive == 'new':
archive = ctx.client.create_archive(vault)['id'] archive = client.create_archive(vault)['id']
printer("Uploading {} files ({}) to new archive: {}/{}", len(uploads), hbytes(total), vault, archive) printer("Uploading {} files ({}) to new archive: {}/{}", len(uploads), hbytes(total), vault, archive)
else: else:
printer("Uploading {} files ({}) to archive: {}/{}", len(uploads), hbytes(total), vault, archive) printer("Uploading {} files ({}) to archive: {}/{}", len(uploads), hbytes(total), vault, archive)
...@@ -90,17 +100,22 @@ def command(ctx, args): ...@@ -90,17 +100,22 @@ def command(ctx, args):
pbar = tqdm(total=total, unit='b', unit_scale=True, unit_divisor=1024, dynamic_ncols=True, pbar = tqdm(total=total, unit='b', unit_scale=True, unit_divisor=1024, dynamic_ncols=True,
file=printer.file) file=printer.file)
for i, target in enumerate(sorted(uploads)): try:
file, stat = uploads[target] for i, target in enumerate(sorted(uploads)):
with open(file, 'rb') as fp: file, stat = uploads[target]
line = "[{}/{}] {}".format(i + 1, len(uploads), target[1:]) with open(file, 'rb') as fp:
if pbar: line = "[{}/{}] {}".format(i + 1, len(uploads), target[1:])
pbar.write(line) if pbar:
read = fp.read pbar.write(line)
chunks = iter(lambda: read(1024*8), b'') read = fp.read
chunks = (chunk for chunk in chunks if not pbar.update(len(chunk))) chunks = iter(lambda: read(1024*8), b'')
fp = chunks chunks = (chunk for chunk in chunks if not pbar.update(len(chunk)))
else: fp = chunks
printer(line) else:
ctx.client.put_file(vault, archive, target, fp) printer(line)
client.put_file(vault, archive, target, fp)
finally:
if pbar:
pbar.close()
printer("Done") printer("Done")
import os import os
import urllib.parse import urllib.parse
from contextlib import contextmanager
from . import printer, CliError from . import printer, CliError
from ._utils import walk_up from ._utils import walk_up
...@@ -13,50 +14,103 @@ class CliContext: ...@@ -13,50 +14,103 @@ class CliContext:
""" Search for config files and archive directory in or above the current working directory. """ """ Search for config files and archive directory in or above the current working directory. """
def __init__(self, workdir="."): def __init__(self, workdir="."):
self.connected = {}
self.workdir = os.path.abspath(workdir) self.workdir = os.path.abspath(workdir)
self.archive_root = None self.archive_root = None
self.config = None self._config = None
self._client = None
home = os.path.abspath(os.path.expanduser("~")) def load_config(self, fname):
self._config = Config(fname)
@property
def config(self):
if not self._config:
self._find_config()
if not self._config:
raise CliError("No config found (see `init` command)")
return self._config
def _find_config(self):
for path in walk_up(self.workdir): for path in walk_up(self.workdir):
if not self.archive_root: for name in CONFIG_NAMES:
if os.path.exists(os.path.join(path, METADIR_NAME)): cfile = os.path.join(path, name)
printer.vv("Found archive directory: {}", path) if os.path.exists(cfile):
self.archive_root = path printer.vv("Found config file: {}", cfile)
if not self.config: self._config = Config(cfile)
for name in CONFIG_NAMES: return
cfile = os.path.join(path, name)
if os.path.exists(cfile): def _find_root(self):
printer.vv("Found config file: {}", cfile) for path in walk_up(self.workdir):
self.config = Config(cfile) if os.path.exists(os.path.join(path, METADIR_NAME)):
break printer.vv("Found archive directory: {}", path)
if path == home: self.archive_root = path
return return
def resolve(self, ref):
""" Return (client, vault, archive, file) from a reference string
a(/f) -> defaultServer, defaultVault, a, (/f)
/v(/a(/f)) -> defaultServer, v, (a), (/f)
http(s)://server.tpl/v3/v(/a(/f)) -> server, v, (a), (/f)
"""
oref = ref
if ref.startswith("http://") or ref.startswith("https://"):
url, v3, rest = ref.partition("/v3/")
if not v3: raise ValueError("Not a CDSTAR url: " + oref)
client = self.connect(url + v3)
ref = '/' + rest
else:
client = self.connect(self.config['server'])
if ref.startswith('/'):
vault, _, ref = ref[1:].partition('/')
if not vault:
raise ValueError("Not a valid target: " + oref)
else:
vault = self.config['vault']
archive, slash, file = ref.partition("/")
if slash:
file = slash + file
return client, vault, archive, file
def connect(self, server):
if server not in self.connected:
url = urllib.parse.urlsplit(server)
if url.username:
auth = (url.username, url.password or self._ask_pass(url))
client = CDStar(server, auth=auth)
else:
client = CDStar(server)
self.connected[server] = client
return self.connected[server]
@property @property
def default_vault(self): def default_vault(self):
if not self.config or 'vault' not in self.config: if 'vault' not in self.config:
raise CliError("No default vault configured") raise CliError("No default vault configured")
return self.config['vault'] return self.config['vault']
@property def _ask_pass(self, url):
def client(self): # Enter password for {url.scheme}://{url.netloc}/{url.path} (user={url.username})
if self._client: raise RuntimeError("Asking for password not implemented yet")
return self._client
if not self.config:
raise CliError("Unable to connect: No config found (see `init` command)") class VerboseTx:
server = self.config["server"] def __init__(self, tx):
url = urllib.parse.urlparse(server) self.tx = tx
if url.username:
auth = (url.username, url.password or self._ask_pass()) def __enter__(self):