HTCondors

HTCondor, often referred to as Condor or HTCondors, is an open-source workload management system designed for high-throughput computing across clusters and grids. It provides a distributed scheduling and resource-management framework that enables large numbers of tasks to run on heterogeneous hardware. The project originated at the University of Wisconsin–Madison and has become widely used in academia and industry for managing batch workloads and scientific computations.

HTCondor operates as a set of coordinated daemons that manage and allocate resources. The central manager comprises a collector and negotiator that maintain a pool of machines and match jobs with available resources. Submission hosts run condor_schedd to manage the queue, while execution hosts run condor_startd to advertise capacity. Additional daemons track and control job execution and facilitate communication between components.

Key features include policy-driven scheduling, backfilling to maximize utilization, fault tolerance and checkpointing, and the ability to preempt lower-priority tasks when higher-priority work arrives. HTCondor supports heterogeneous resources such as multicore CPUs and GPUs, and can integrate with virtualization or container technologies to run jobs in isolated environments. It also provides data staging, file transfer, monitoring, and accounting capabilities.

HTCondor is commonly used for high-throughput scientific workloads, parameter sweeps, data analysis, and workflow management. It scales from a single cluster to multi-site deployments and can interoperate with other batch systems and provisioning tools to manage resources across administrative boundaries.

HTCondor is released under a permissive open-source license and is maintained by the University of Wisconsin–Madison and a broad community of contributors. Documentation and source code are available from the project’s website and repositories.