Gitlab Community Edition Instance

Skip to content

GitLab

Commit fa779ab0 authored by mhellka's avatar mhellka
Browse files

TUS: Calculate TUS handle expiration based on last modification time.

parent 15854854
Pipeline #292152 canceled with stages
in 9 minutes and 31 seconds
Hide whitespace changes
Inline Side-by-side
cdstar-plugins-pom/cdstar-tus/README.adoc
View file @ fa779ab0
......@@ -3,12 +3,11 @@
This `TusPlugin` installs a https://tus.io/[tus.io] compatible REST endpoint to upload temporary files, and a way for other APIs to reference these files via server-side data streams. This helps clients to upload large files over unreliable network connections, or parallelize uploads of multiple files for the same archive.
TIP: Tus will NOT improve upload speed or throughput over stable network connections. The fastest and most efficient way to upload large files to cdstar is via <<putFile>>. The best way to upload many small files to cdstar is via <<updateArchive>>. Only use tus if uploads need to be resumable or repeatable.
TIP: TUS will NOT improve upload speed or throughput over stable network connections. The fastest and most efficient way to upload large files to cdstar is via <<putFile>>. The best way to upload many small files to cdstar is via <<updateArchive>>. Only use TUS if uploads need to be resumable or you want to import the same file multiple times.
## Configuration
There is currently no configuration for this plugin. Uploads will be placed into `${path.var}/tus/` and deleted after 24h.
There is currently no configuration for this plugin. Uploads will be placed into `${path.var}/tus/`.
.Example Configuration
[source,yaml]
......@@ -27,7 +26,9 @@ plugin.tus.class: TusPlugin
## Usage
The https://tus.io/[tus.io] compatible REST endpoint is reachable under `/tus` at the root-level of the service (next to `/v3`, not under it). After creating an upload target and uploading data following tus protocol, the temporary resource can be referenced as `tus:<tusId>`, where `<tusId>` is the last part of the tus resource URI. For example, if your tus upload target was `/tus/24e533e`, then the internal reference to this resource would be `tus:24e533e`.
The https://tus.io/[tus.io] compatible REST endpoint is reachable under `/tus` at the root-level of the service (not `/v3/tus` but just `/tus`). After creating a TUS handle and uploading data following TUS protocol, the temporary file can be referenced as `tus:<tusId>`, where `<tusId>` is the last part of the TUS handle. For example, if your TUS handle was `/tus/24e533e`, then the internal reference to this resource would be `tus:24e533e`.
Currently only the <<createArchive>> and <<updateArchive>> support server-side imports via the `fetch:<target>` functionality. For example, to import a completed TUS upload into an archive, you would send `fetch:/path/to/target.file=tus:24e533e` as a POST form parameter. Note that the digests must still be computed, so a fetch may take just as long as uploading the file directly. TUS usually does not improve overall throughput, but may improve reliability of large-file uploads over unreliable network connections. Use it wisely.
Currently only the <<createArchive>> and <<updateArchive>> support server-side imports via the `fetch:<target>` functionality. For example, to import an existing tus resource into an archive, you would send `fetch:/path/to/target.file=tus:24e533e` as a POST form parameter. Note that the digests must still be computed for server-side imports, so a fetch may take just as long as uploading the file directly. Tus usually does not improve overall throughput, but may improve reliability of large-file uploads over unreliable network connections. Use it wisely.
Incomplete TUS handles that do not see any new data will expire after 2 hours. Once complete, the TUS handle can be referenced for another 24 hours before it expires. Handles that are not needed anymore can (and should) be deleted faster with a single `DELETE` request to the TUS handle.
cdstar-plugins-pom/cdstar-tus/src/main/java/de/gwdg/cdstar/rest/ext/tus/TusPlugin.java
View file @ fa779ab0
package de.gwdg.cdstar.rest.ext.tus;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.TimeUnit;
import com.codahale.metrics.Gauge;
......@@ -11,7 +10,6 @@ import de.gwdg.cdstar.runtime.Plugin;
import de.gwdg.cdstar.runtime.RuntimeContext;
import de.gwdg.cdstar.runtime.listener.RuntimeListener;
import de.gwdg.cdstar.runtime.services.CronService;
import de.gwdg.cdstar.runtime.services.FeatureRegistry;
import de.gwdg.cdstar.runtime.services.PoolService;
/**
......@@ -53,9 +51,6 @@ public class TusPlugin implements RuntimeListener {
service.start(ctx.lookupRequired(PoolService.class).getNamedPool("tus"));
ctx.lookup(FeatureRegistry.class)
.ifPresent(fr -> fr.addFeature("tus", "1.0"));
ctx.lookupRequired(CronService.class)
.scheduleWithFixedDelay(service::cleanupExpiredUploads, 60, 60, TimeUnit.SECONDS);
}
......
cdstar-plugins-pom/cdstar-tus/src/main/java/de/gwdg/cdstar/rest/ext/tus/TusService.java
View file @ fa779ab0
......@@ -28,7 +28,10 @@ public class TusService {
public static final String pluginName = "tus-upload";
static final Logger log = LoggerFactory.getLogger(TusService.class);
Duration defaultExpire = Duration.ofHours(24);
// Both timeouts are relative to the last chunk uploaded.
Duration expireIncompleteAfter = Duration.ofHours(2);
Duration expireFinishedAfter = Duration.ofHours(24);
Path basePath;
Map<String, TusUpload> uploads = new ConcurrentHashMap<>();
......@@ -59,7 +62,8 @@ public class TusService {
public void cleanupExpiredUploads() {
final Instant now = Instant.now();
// Collect (and lock) expired chunks in a thread-save way.
// Collect (and lock) expired uploads in a thread-save way.
// Do not expire locked uploads, as they might be in use.
final List<TusUpload> expired = new ArrayList<>();
uploads.values().forEach(upload -> {
if (upload.isExpired(now) && upload.tryLock()) {
......
cdstar-plugins-pom/cdstar-tus/src/main/java/de/gwdg/cdstar/rest/ext/tus/TusUpload.java
View file @ fa779ab0
package de.gwdg.cdstar.rest.ext.tus;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.channels.WritableByteChannel;
import java.nio.file.Files;
......@@ -24,15 +25,15 @@ class TusUpload {
private long length = -1;
private String meta;
private final Instant created;
private Instant lastModified;
TusUpload(TusService service) {
this.service = service;
this.name = Utils.bytesToHex(Utils.randomBytes(16));
this.created = Instant.now();
name = Utils.bytesToHex(Utils.randomBytes(16));
created = lastModified = Instant.now();
}
public TusUpload(TusService service, Path infoFile) throws IOException {
TusUpload(TusService service, Path infoFile) throws IOException {
this.service = service;
var json = SharedObjectMapper.json.readTree(infoFile.toFile());
name = json.path("name").textValue();
......@@ -40,6 +41,7 @@ class TusUpload {
length = length < 0 ? -1 : length;
meta = json.path("meta").textValue();
created = DateTimeFormatter.ISO_INSTANT.parse(json.path("created").textValue(), Instant::from);
lastModified = Files.getLastModifiedTime(Files.exists(getChunkPath()) ? getChunkPath() : infoFile).toInstant();
}
public synchronized void persistInfo() throws IOException {
......@@ -59,8 +61,10 @@ class TusUpload {
}
public synchronized void unlock() {
if(writeChannel != null)
if (writeChannel != null) {
Utils.closeQuietly(writeChannel);
writeChannel = null;
}
locked = false;
}
......@@ -86,7 +90,7 @@ class TusUpload {
public synchronized long getCurrentOffset() {
try {
if(writeChannel != null && writeChannel.isOpen())
if (writeChannel != null && writeChannel.isOpen())
return writeChannel.position();
return Files.size(getChunkPath());
} catch (final IOException e) {
......@@ -94,6 +98,10 @@ class TusUpload {
}
}
public synchronized Instant getLastModified() {
return lastModified;
}
public String getMeta() {
return meta;
}
......@@ -103,8 +111,7 @@ class TusUpload {
}
public Instant getExpiresAt() {
// TODO: Different expiration times for in-progress and completed
return created.plus(service.defaultExpire);
return getLastModified().plus(isComplete() ? service.expireFinishedAfter : service.expireIncompleteAfter);
}
Path getPath() {
......@@ -125,23 +132,53 @@ class TusUpload {
@Override
public String toString() {
return "tus:"+name;
return "tus:" + name;
}
public synchronized WritableByteChannel getWriteChannel() throws IOException {
if(writeChannel != null && writeChannel.isOpen())
if (writeChannel != null && writeChannel.isOpen())
throw new IllegalStateException("Write in progress");
if(!locked)
if (!locked)
throw new IllegalStateException("Upload not locked");
writeChannel = FileChannel.open(getChunkPath(), StandardOpenOption.CREATE, StandardOpenOption.WRITE, StandardOpenOption.APPEND);
return writeChannel;
writeChannel = FileChannel.open(getChunkPath(), StandardOpenOption.CREATE, StandardOpenOption.WRITE,
StandardOpenOption.APPEND);
return new WritableByteChannel() {
WritableByteChannel delegate = writeChannel;
@Override
public boolean isOpen() {
return delegate.isOpen();
}
@Override
public void close() throws IOException {
if (isOpen()) {
synchronized (TusUpload.this) {
if (writeChannel != delegate)
throw Utils.wtf();
lastModified = Instant.now();
Utils.closeQuietly(writeChannel);
writeChannel = null;
}
}
}
@Override
public int write(ByteBuffer src) throws IOException {
synchronized (TusUpload.this) {
lastModified = Instant.now();
}
return delegate.write(src);
}
};
}
public synchronized FileChannel getReadChannel() throws IOException {
if(writeChannel != null && writeChannel.isOpen())
if (writeChannel != null && writeChannel.isOpen())
throw new IllegalStateException("Write in progress");
if(!locked)
if (!locked)
throw new IllegalStateException("Upload not locked");
return FileChannel.open(getChunkPath(), StandardOpenOption.READ);
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment