Queues Available for Colmena
Colmena supports multiple backends for queues that pass data between Thinker and Task Server (see Design)
Python Pipes
PipeQueues
uses Python Pipes transmit data
between two Python processes.
Advantages:
Simple setup (no configuration or other services required)
Very portable. Uses only Python native libraries
Disadvantages:
Small message sizes (<32 MiB)
Thinker and Task Server must be on same system
Only one Thinker and Task server are allowed
Redis
RedisQueues
uses Redis, a high-performance in-memory data store.
Advantages:
Support moderate message sizes (<512 MiB)
Thinker and Task Server can run on different systems
Applications can use multiple Thinkers and Task Servers
Redis server can also serve as a backend for ProxyStore
Disadvantages:
Redis must run as a second service
Redis is difficult to install on some OSs or architectures
Open ports or SSH tunnels may be required if Redis on separate host from Task Server/Thinker