Learning Higher-Order Logic Programs From Failures         [Extended Version]

Inductive Logic Programming, abbreviated ILP, Muggleton (1991); Nienhuys-Cheng et al. (1997) is a form of symbolic machine learning which learns a logic program from background knowledge ($\mathit{BK}$) predicates and sets of positive and negative example runs of the goal program.

Naively, learning a logic program which takes a positive integer $n$ and returns a list of list of the form $[[1],[1,2],$ $\cdots,[1,\cdots,n]]$ would not come across as a formidable learning task. A logic program is easily constructed using conventional higher-order (HO) definitions.

The first $\mathtt{iterate}$¹¹ 1 See Appendix of arXiv:2112.14603 for HO definitions. produces the list $[1,\cdots,N]$ and $\mathtt{map}$ applies a functionally equivalent $\mathtt{iterate}$ to each member of $[1,\cdots,N]$, thus producing the desired outcome. However, this seemingly innocuous function requires 25 literals spread over five clauses when written as a function-free, first-order (FO) logic program, a formidable task for most if not all existing FO ILP approaches Cropper et al. (2022).

Excessively large $\mathit{BK}$ can, in many cases, lead to performance loss Cropper (2020); Srinivasan et al. (2003). In contrast, adding HO definitions increases the overall size of the search space, but may result in the presence of significantly simpler solutions (see Figure 1). Enabling a learner, with a strong bias toward short solutions, with the ability to use HO definitions can result in improved performance. We developed an HO-enabled Popper Cropper and Morel (2021a) (Hopper), a novel ILP system designed to learn optimally short solutions. Experiments show significantly better performance on hard tasks when compared with Popper and the best performing HO-enabled ILP system, Metagol_HO Cropper et al. (2020). See Section 4.

Existing HO-enabled ILP systems are based on Meta-inter-pretive Learning (MiL) Muggleton et al. (2014). The efficiency and performance of MiL-based systems is strongly dependent on significant human guidance in the form of metarules (a restricted form of HO horn clauses). Choosing these rules is an art in all but the simplest of cases. For example, $\mathtt{iterate}$, being ternary (w.r.t. FO arguments), poses a challenge for some systems, and in the case of HEXMIL_HO Cropper et al. (2020), this definition cannot be considered as only binary definitions are allowed (w.r.t. FO arguments).

Limiting human participation when fine-tuning the search space is an essential step toward strong symbolic machine learning. The novel Learning from Failures (LFF) paradigm Cropper and Morel (2021a), realized through Popper, prunes the search space as part of the learning process. Not only does this decrease human guidance, but it also removes limitations on the structure of HO definitions allowing us to further exploit the above-mentioned benefits.

Integrating HO concepts into MiL-based systems is quite seamless as HO definitions are essentially a special type of metarule. Thus, HO enabling MiL learners requires minimal change to the theoretical foundations. In the case of LFF learners, like Popper, the pruning mechanism influences which HO definitions may be soundly used (See page 813 of Cropper and Morel (2021a)).

We avoid these soundness issues by indirectly adding HO definitions. Hopper  uses FO instances of HO definitions each of which is associated with a set of unique predicates symbols denoting the HO arguments of the definition. These predicates symbols occur in the head literal of clauses occurring in the candidate program iff their associated FO instance occurs in the candidate program. Thus, only programs with matching structure may be pruned. We further examine this issue in Section 3 and provide a construction encapsulating the accepted class of HO definitions.

Succinctly, we work within the class of HO definitions that are monotone with respect to subsumption and entailment; $p_{1}\leq_{\theta}\hskip-4.49997pt(\models)\ p_{2}\Rightarrow H(p_{1})\leq_{\theta}\hskip-3.00003pt(\models)\ H(p_{2})$ where $p_{1}$ and $p_{2}$ are logic programs, and $H(\cdot)$ is an HO definition incorporating parts of $p_{1}$ and $p_{2}$. Similar to classes considered in literature, our class excludes most cases of HO negation (see Section 3.4). However, our framework opens the opportunity to invent HO predicates during learning (an important open problem), though this remains too inefficient in practice and is left to future work.

The authors of Cropper et al. (2020) (Section 2) provide a literature survey concerning the synthesis of Higher-Order (HO) programs and, in particular, how existing ILP systems deal with HO constructions. We provide a brief summary of this survey and focus on introducing the state-of-the-art systems, namely, HO extensions of Metagol Cropper and Muggleton (2016) and HEXMIL Kaminski et al. (2018). Also, we introduce Popper Cropper and Morel (2021a), the system Hopper is based on. For interested readers, a detailed survey of the current state of ILP research has recently been published Cropper et al. (2022).

We provide a brief overview of logic programming. Our exposition is far from comprehensive. We refer the interested reader to a more detailed source Lloyd (1987).

A possible, albeit very weak, program synthesizer is an enumeration procedure that orders all possible programs constructible from the $\mathit{BK}$ by size, testing each until a solution is found. In Cropper and Morel (2021a), this procedure was referred to as Enumerate. Popper extends Enumerate by pruning the hypothesis space based on previously tested programs.

The pruning mechanism will never prune the shortest solution. Thus, the important question when evaluating Popper, and Hopper, is not if Popper will find a solution, nor is it a high-quality solution, but rather how long it takes Popper to find the solution. An extensive suite of experiments was presented in Cropper and Morel (2021a), illustrating that Popper  outperforms Enumerate and existing ILP systems.

One way to improve the performance of LFF-based ILP systems, like Popper, is to introduce techniques that shorten or simplify the solution. The authors of Cropper et al. (2020), in addition to introducing Metagol_HO and HEXMIL_HO, provided a comprehensive suite of experiments illustrating that the addition of HO predicates can improve accuracy and, most importantly, reduce learning time. Reduction in learning times results from a reduction in the complexity/size of the solutions.

The experiments in Cropper and Morel (2021a) thoroughly cover scalability issues and learning performance on simple list transformation tasks, but do not cover performance on complex tasks with large solutions. The experiments presented in Cropper et al. (2020) illustrate performance gains when a HO library is used to solve many simple tasks and how the addition of HO predicates allows the synthesis of relatively complex predicates such as dropLast. When the solution is large Popper’s performance degrades significantly. When the solution requires complex interaction between predicates and clauses it becomes exceedingly difficult to find a set of metarules for Metagol_HOwithout being overly descriptive or suffering from long learning times.

Our experiments illustrate that the combination of Popper and HO predicates Cropper et al. (2020) significantly improves Popper’s performance at learning complex programs. Similar to Cropper and Morel (2021a), we use predicate declarations, i.e. body_pred(head,2), type declarations, i.e. type(head,(list,element)), direction declarations, i.e. direction(head,(in,out)), and the parameters required by Popper’s search mechanism, max_var, max_body, and max_clauses.

We reevaluated 7 of the tasks presented in Cropper and Morel (2021a) and 2 presented in Cropper et al. (2020). Additionally, we added 8 list manipulation tasks, 3 tree manipulation tasks, and 2 arithmetic tasks (separated by type in Table 1). Our additional tasks are significantly harder than the tasks evaluated in previous work.

For each task, we guarantee that the optimal solution is present in the hypothesis space and record how long Popper and Hopper take to find it. We ran Popper using optimal settings and minimal $\mathit{BK}$. In some cases, the tasks cannot be solved by Popper without Predicate Invention (See Column PI? of Table 1), i.e. a explanatory hypothesis which is both accurate and precise requires auxiliary concepts.

We ran Hopper in two modes, Column Hopper  concerns running Hopper with the same settings and a superset of the $\mathit{BK}$ used by Popper (minus constructions used to force invention), while Column Hopper (Opt) concerns running Hopper with optimal settings and minimal $\mathit{BK}$. For Popper and Hopper, settings such as max_var significantly impact performance. Both systems search for the shortest program (by literal count) respecting the current constraints. Note, that hypothesizing a program with an additional clause w.r.t the previously generated programs requires increasing the literal count by at least two. Thus, the current search procedure avoids such hypotheses until all shorter programs have been pruned or tested. The parameters max_var and max_body have a significant impact on the size of the single clause hypothesis space. Given that the use of HO definitions always requires auxiliary clauses, using large values, for the above-mentioned parameter, will hinder their use. This is why Hopper performs significantly better post-optimization. Using a comparably large $\mathit{BK}$ incurs an insignificant performance impact compared to using unintuitively large parameter settings (see Proposition 1, page 14 Cropper and Morel (2021a)).

The predicates used for a particular task are listed in Column HO-Predicate of Table 1. Popper and Hopper timed out (300 seconds elapsed) when large clauses or many variables are required. Timing out means the optimal solution was not found in 300 seconds. Given that we know the solution to each task, Column #Literals provides the size of the known solution, not the size of the non-optimal solution found by the system in case of timeout.

Concerning the Optimizations, these runs of Hopper closely emulate how such a system would be used as it is pragmatic to search assuming smaller clauses and fewer variables are sufficient and expand the space as needed. Popper’s and Hopper’s performance degrades by just assuming the solution may be complex. For findDup, Hopper found the FO solution.

Overall, this optimization issue raises a question concerning the search mechanism currently used by both Popper and Hopper. While HO solutions are typically shorter than the corresponding FO solution, this brevity comes at the cost of a complex program structure. This trade-off is not considered by the current implementations of the LFF paradigm. Investigating alternative search mechanisms and optimality conditions (other than literal count) is planned future work.

We attempted to solve each task using Metagol_HO. Successful learning using Metagol_HO is highly dependent on the choice of the metarules. To simplify matters, we chose metarules that mimic the clauses found in the solution. In some cases, this requires explicitly limiting how certain variables are instantiated by adding declarations, i.e. metagol:type(Q,2,head_pred), to the body of a metarule (denoted by metatype in Table 1). This amounts to significant human guidance, and thus, both simplifies learning and what we can say comparatively about the system. Hence we limit ourselves to indicating success or failure only.

Under these experimental settings, every successful task was solved faster by Metagol_HO than Hopper with optimal settings. Relaxing restrictions on the metarule set introduces a new variable into the experiments. Choosing a set of metarules that is general and covers every task will likely result in the failure of the majority of tasks. Some tasks require splitting rules such as $P(A,B)\mbox{:-}~Q(A,B),R(A,B)$ which significantly increase the size of the hypothesis space. Choosing metarules per task, but without optimizing for success, leaves the question, which metarules are appropriate/acceptable for the given task? While this is an interesting question Cropper and Tourret (2020), the existence problem of a set of general metarules over which Metagol_HO’s performance is comparable to, or even better than, Hopper’s only strengthens our argument concerning the chosen experimental setting as one will have to deduce/design this set.

We extended the LFF-based ILP system Popper to effectively use user-provided HO definitions during learning. Our experiments show Hopper (when optimized) is capable of outperforming Popper on most tasks, especially the harder tasks we introduced in this work (Section 4). Hopper requires minimal guidance compared to the top-performing MiL-based ILP system Metagol_HO. Our experiments test the theoretical possibility of Metagol_HO finding a solution under significant guidance. However, given the sensitivity to metarule choice and the fact that many tasks have ternary and even 4-ary predicates, it is hard to properly compare these approaches.

We provide a theoretical framework encapsulating the accepted HO definitions, a fragment of the definitions monotone over subsumption and entailment, and discuss the limitations of this framework. A detailed account is provided in Section 3. The main limitation of this framework concerns HO-negation which we leave to future work. Our framework also allows for the invention of HO predicates during learning through constructions of the form $ho(P,Q,x,y)\mbox{:-}~P(Q,x,y)$. We can verify that Hopper can, in principal, finds the solution, but we have notsuccessfully invent an HO predicate during learning. We plan for further investigation of this problem.

As briefly mentioned, Hopper was tested twice during experimentation due to the significant impact system parameters have on its performance. This can be seen as an artifact of the current implementation of LFF which is bias towards programs with fewer, but longer, clauses rather than programs with many short clauses. An alternative implementation of LFF taking this bias into account, together with other bias, is left to future investigation.

We would like to thank Rolf Morel and Andrew Cropper for their thorough commentary which helped us greatly improve a preliminary version of this paper.