982 error this iri is already being processed (#1697) * Add expiration parameter to Semaphore After the expiration, the semaphore is unlocked. This allows dead processes (e.g. by kill -9) to release the lock again. * Namespace the redis semaphore at ontohub:semaphore. * Code style. * Code style: use class << self block. * Add concurrency testing gem. * Completely revamp Semaphore * Put it into the same redis-namespace as sidekiq. * Add method to check if a lock is closed. * Namespace semaphores even deeper. Checking if a lock is closed is not an atomic operation. To make it atomic, we put the check into a mutexed part. In order to really exclude the locking and unlocking from this mutexed check, these two operations must be mutexed as well with the same key (LOCK_ACTION_KEY). Unfortunately, using many threads/processes in an example makes the test suite hang. This is because of celluloid. Maybe we can unskip the example after upgrading to sidekiq 4 and dropping sidetiq. * Add settings keys for Redis. * Use redis namespace setting for Semaphore. * Share redis connections between Sidekiq and Semaphore When all connections are in use, any Redis call will block until a connection is freed. Sidekiq is now namespaced deeper in order to prevent conflicts with the semaphore. * Fix locked? and perform_exclusively signatures. * Add lock and unlock methods. * Use separate redis namespaces for our environments. * Use Semaphore only if not testing * Use Semaphore when parsing an Ontology * Remove not needed method perform_async_on_queue. * Refactor all the Worker classes for parsing Now, all comes down to the new OntologyParsingWorker. It performs on a queue of triples containing the version id, options for the version and the try_count (number of the next parsing attempt). If parsing a version fails, it is put back to the queue (at the end), with an incremented try_count. If this happens too often for the same version, it is given to the SequentialOntologyParsingWorker that only parses one OntologyVersion at a time. The OntologyParsingPriorityWorker is a more elegant way to use the priotitized queue. All the background parsing is now performed via the OntologyParsingWorker or its subclasses. * Be explicit about the namespace. * Obey Hound.