Performance Test of Celery Prototype
In a peer-coding approach, test the performance of the Celery Prototype on a GWDG Cloud Server (use a large and a small one, just for fun). Run a complex (i.e. a resource hungry) TOS Job on it; either try to get it to crash and check the results or check how Celery handles high loads.
DoD
- performance test conducted
- crash messages stored
- test of different result backends (PostgresSQL, CouchDB)
Activity
added nextWeek tech::Weighted labels
added tech::ToDo label and removed tech::Weighted label
removed nextWeek label
added tech::Doing label and removed tech::ToDo label
Performance Test
The solution using Containers proved to be rather robust. Even if Containers are removed via Docker, the Celery Worker takes note of that reports it back to the Celery farmer. Thats neat
All in all, the performance was a disaster. A Job that takes about 5 Minutes locally (on 32GB RAM) took about 5 hours on the server (4GB, although there were 4 Jobs running at the same time). I assume this can be optimized largely by modifying the Docker Resource settings. Also, the Jobs did not cancel or error out, they were running solidly. Moreover, Celery gives a CLI to check on those Jobs, which is nice.
Finally, I encountered an issue with multiple Jobs being queued by the workers. I should look into this, maybe there is some kind of timeout that was reached.
Crash messages stored
Yep, they are. They can be found in the result backend. The schema for that on PostgreSQL is very simple and can be read by anyone in addition to the Celery CLI possibilities to query the results.
Backends
PostgreSQL worked without any issues. CouchDB seems to be broken right now and cannot be tested
Next steps
- Improve performance of Docker Containers (the Cloud Server was not on high loads while checking periodically)
- Check configuration possibilities of Celery Workers
- Re-visit CouchDB Result Backend when the Celery Issue is fixed
I will create a new Issue for that
mentioned in issue #17 (closed)