Queues Available for Colmena

Colmena supports multiple backends for queues that pass data between Thinker and Task Server (see Design)

Python Pipes

PipeQueues uses Python Pipes transmit data between two Python processes.

Advantages:

  • Simple setup (no configuration or other services required)

  • Very portable. Uses only Python native libraries

Disadvantages:

  • Small message sizes (<32 MiB)

  • Thinker and Task Server must be on same system

  • Only one Thinker and Task server are allowed

Redis

RedisQueues uses Redis, a high-performance in-memory data store.

Advantages:

  • Support moderate message sizes (<512 MiB)

  • Thinker and Task Server can run on different systems

  • Applications can use multiple Thinkers and Task Servers

  • Redis server can also serve as a backend for ProxyStore

Disadvantages:

  • Redis must run as a second service

  • Redis is difficult to install on some OSs or architectures

  • Open ports or SSH tunnels may be required if Redis on separate host from Task Server/Thinker