Concept: robustness optimization of job queueing and running

As the number of TOS Jobs grows, one Container is unable to handle the load of multiple TOS Jobs (they are way too performance hungry, being Java and all...).

A natural way would be to re-visit Celery and create a concept on how to implement celery to the TOS Agent. Note that this could mean the the way to upload Jobs may change dramatically.

@msuhr1 you checked out Celery a while back, I guess that you could be the right person to verify this weight

There's a significant amount of uncertainty involved with this question so I agree on your estimate.

Any specific reason that you expect Celery might reduce the performance issues?

Celery is much more established than the solution I created myself. It may not increase performance per se, but it should add a lot more robustness towards a high number of jobs. Ultimately, I expect Celery to provide a better possibility to just add resources to and allow a performance increase that way.

But you are right, performance increase is pretty much wrong here

added nextWeek label

added tech::Weighted label

changed title from Concept: performance optimization of job queueing and running to Concept: robustness optimization of job queueing and running

Moving to tech::ToDo in order to speed up the implementation of it

added tech::ToDo label and removed tech::Weighted label

removed nextWeek label

added tech::Doing label and removed tech::ToDo label

mentioned in merge request !32 (closed)

created merge request !32 (closed) to address this issue

assigned to @portal.mparcia

Current state: first approach seems to work pretty easily: https://gitlab.gwdg.de/medinfpub/umg-medic/technik/celery-docker-test

I run it on a GWDG cloud server for now and will add some explanation to it soon. Next up: testing docker support and adding that to this barebone.

I implemented a prototype and added some documentation also. You can test and review the basic approach in this repository.

The code should not be of main interest here. Check the basic concept I applied to distribute TOS Jobs as tasks onto Celery workers and all the services I added to the agent.

Could we maybe have a semi-spontaneous developer meeting tomorrow where you talk the rest of us through the changes and concepts? Some time in the afternoon?

Sure thing, I will schedule a meeting.

added tech::Review label and removed tech::Doing label

Added Merge Request: https://gitlab.gwdg.de/medinfpub/umg-medic/technik/celery-docker-test/-/merge_requests/1

As discussed, @tobias.zimmer, check the merge request and the comments to hopefully get some insight on how the prototype works. Feel free to create an own merge request based on the MR in order to refactor some of the code.

Concept as is is done, medinfpub/umg-medic/technik/celery-docker-test!1 (closed) is closed aswell. Further investigation will be done in #17 (closed)

Closing

closed