Check that index can return in get_actual_variable_range()

Some recent changes were made to remove the explicit dependency on
btree indexes in some parts of the code.  One of these changes was
made in commit 9ef1851, which allows non-btree indexes to be used
in get_actual_variable_range().  A follow-up commit ee1ae8b fixes
the cases where an index doesn’t have a sortopfamily as this is a
prerequisite to be used in get_actual_variable_range().

However, it was found that indexes that have amcanorder = true but do
not allow index-only-scans (amcanreturn returns false or is NULL) will
pass all of the conditions, while they should be rejected since
get_actual_variable_range() uses the index-only-scan machinery in
get_actual_variable_endpoint().  Such an index might cause errors like

    ERROR:  no data returned for index-only scan

during query planning.

The fix is to add a check in get_actual_variable_range() to reject
indexes that do not allow index-only scans.

Author: Maxime Schoemans <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/20ED852A-C2D9-41EB-8671-8C8B9D418BE9%40enterprisedb.com

Author: petere

src/backend/utils/adt/selfuncs.c

@@ -6574,6 +6574,13 @@ get_actual_variable_range(PlannerInfo *root, VariableStatData *vardata,
 		if (index->hypothetical)
 			continue;
 
+		/*
+		 * get_actual_variable_endpoint uses the index-only-scan machinery, so
+		 * ignore indexes that can't use it on their first column.
+		 */
+		if (!index->canreturn[0])
+			continue;
+
 		/*
 		 * The first index column must match the desired variable, sortop, and
 		 * collation --- but we can use a descending-order index.