Improve PostgreSQL replication lag detection

[jsosic]

In some cases master can show pg_last_xact_replay_timestamp() from past, which can cause the exporter to show ever-growing value for the lag. This fixes the issue.

I've got the idea for fix from:

• mailing list
• DataDog bug fix

Example (from master):

[local]:5432 postgres@postgres=# SELECT EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp())) as lag;
+----------------+
|      lag       |
+----------------+
| 3567749.711759 |
+----------------+
(1 row)

Same master, fixed query:

[local]:5432 postgres@postgres=# SELECT CASE WHEN NOT pg_is_in_recovery() THEN 0 ELSE GREATEST (0, EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp()))) END AS lag;
+-----+
| lag |
+-----+
|   0 |
+-----+
(1 row)

Master recovery status:

[local]:5432 postgres@postgres=# SELECT pg_is_in_recovery();
+-------------------+
| pg_is_in_recovery |
+-------------------+
| f                 |
+-------------------+
(1 row)

Streaming slave:

[local]:5432 postgres@postgres=# SELECT pg_is_in_recovery();
+-------------------+
| pg_is_in_recovery |
+-------------------+
| t                 |
+-------------------+
(1 row)

[local]:5432 postgres@postgres=# SELECT CASE WHEN NOT pg_is_in_recovery() THEN 0 ELSE GREATEST (0, EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp()))) END AS lag;
+---------+
|   lag   |
+---------+
| 0.31566 |
+---------+
(1 row)

Paused slave:

[local]:5432 postgres@postgres=# SELECT CASE WHEN NOT pg_is_in_recovery() THEN 0 ELSE GREATEST (0, EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp()))) END AS lag;
+--------------+
|     lag      |
+--------------+
| 28053.150006 |
+--------------+
(1 row)


[local]:5432 postgres@postgres=# SELECT pg_is_in_recovery();
+-------------------+
| pg_is_in_recovery |
+-------------------+
| t                 |
+-------------------+
(1 row)

From the documentation:

True if recovery is still in progress.

[coveralls]

Coverage remained the same at 64.7% when pulling dc87b6e on jsosic:fix-psql-replication-lag into e2df41f on wrouesnel:master.