0

I have 3 servers setup: 1 primary, 2 backups. I used the following configuration:

backend pg_production_backend
    option pgsql-check user pg_user
    server primary pghost.primary:5432 check on-marked-down shutdown-sessions on-marked-up shutdown-backup-sessions
    server secondary pghost.secondary:5432 check backup on-marked-down shutdown-sessions on-marked-up shutdown-backup-sessions
    server tertiary pghost.tertiary:5432 check backup on-marked-down shutdown-sessions on-marked-up shutdown-backup-sessions

It works correctly when primary goes down, and connections will go to the first available backup; and connections return to primary when it comes back on.

The problem occurs when primary is down, and secondary is down. Connections will goto tertiary; when secondary comes back online; the connected clients remained on tertiary, and new connections go to secondary.

How to force all connections only goto tertiary or secondary while primary is still down?

Improve this question

2 Answers 2

Reset to default
0

This is how haproxy is designed to work. It doesn't know what state the sessions are in. Idle? Waiting for a result? In the middle of a transaction?

You need to recycle the connections at the client side.

Alternatively use proxysql (do be aware that this has a bigger performance overhead than haproxy).

Improve this answer
4
  • It should not care what state the sessions are in; it should just disconnect all clients and force them to reconnect to the higher tier (or weight) servers. I also tried setting the weights but it made no difference
    – Duy Vu
    Aug 24 at 0:57
  • "It should not care what state the sessions are in" - OMG! YES IT MUST.
    – symcbean
    Aug 27 at 19:45
  • Then how does it switch to non-backup servers when it is back online? Does it care about the state of sessions?
    – Duy Vu
    Aug 29 at 1:05
  • Personally, I use an iptables rule to block new connections on the host I want to isolate. IF you've configured the client end correctly, then the existing connections will be lost through normal recycling over a period. When the connections have fully migrated, I remove the iptables rule.
    – symcbean
    Aug 29 at 12:44
0

You can set a server into the "DRAIN" state, which refuses to take new connections. If you don't have any persistent/sticky rules, all connections should drain from the server fast. You can do that manually in the haproxy admin stats dashboard or automatically by implementing your own haproxy agent.

For example, as long as your secondary is available, your tertiary server could go into the "DRAIN" state and thus is up, but does not allow new connections. This works for sticky-sessions, too, but will take a lot longer, because the session needs to expire first.

If you have persistent settings, and you really want to get rid of the users fast, set the instance into "MAINT" mode (maintenance), and set it back to "UP" again afterwards. But this will cause issues for your users, since the persistent session is gone!

Edit: You can also add the non-stick option to your tertiary server:

Never add connections allocated to this sever to a stick-table. This may be used in conjunction with backup to ensure that stick-table persistence is disabled for backup servers.

If your secondary or primary server goes up again, the connection to them is valued higher by haproxy and the clients get redirected. This can also cause issues to your session persistence!

Related: https://stackoverflow.com/questions/50408562/haproxy-prevent-stickiness-to-a-backup-server

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .