I’ve got a web application that’s currently of several different services, 1 instance each.
My current objective is to scale some of my services to make sure fail over. 1 huge problem I foresee is that a service may execute the Exact Same job with all the instances, here is one example:
I’ve got one service which sends a mail composed of different data grabbed from the net. The email is programmed by the user, either persistent or one time. The planned email is then placed in database with the a datetime field saying when this ought to be done. My solutions are polling every x minutes to see if they have something to do, should they discover something to be done they mark the job as managed and goes on doing it. The problem I face is really between the time that the ceremony grabs the data from the database and the time the upgrade is effective other cases may have grabbed the”bogus” information and send another mail.
This may go the same way for another”observer” service that waits for incoming mails within a single inbox. From polling the inbox and deleting the email once handled, the exact identical race condition can create 2 or more instances handle the received mail.
This question have one part of my issue and the proposed solutions would be good for me. But this, it makes the assumption that there is only one”watcher”.
My other hypothesis is to have an elected master, or even any communication channel between different instances of an agency to get just one of them send the message to the job queue…
I would be delighted to learn how this type of topic is usually handled, astonishingly enough I could not find much info about that, or did not find the right search phrases to look for this.