Several changes

Dropped default number of puzzles to 10
Simplified to a single puzzles.json outfile (removed ProblemSet class)
Renamed codex.py to human_eval.py
Added many human_eval puzzles
Renamed class Problem to PuzzleGenerator
Removed ability to make puzzles with Set solutions
All puzzles now have bool, float, int, str, or List[...] type sols

Author: akalai

README.md

@@ -72,7 +72,7 @@ For more information on the motivation and how programming puzzles can help AI l
 the paper:  
 *Programming Puzzles*, by Tal Schuster, Ashwin Kalyan, Alex Polozov, and Adam Tauman Kalai. 2021 (Link to be added shortly)  
 
-# [Click here to browse the puzzles](/problems/README.md)
+# [Click here to browse the puzzles](/puzzles/README.md)
 
 The problems in this repo are based on:
 * Wikipedia articles about [algorithms](https://en.wikipedia.org/wiki/List_of_algorithms), [puzzles](https://en.wikipedia.org/wiki/Category:Logic_puzzles),
@@ -94,7 +94,7 @@ and see how your programming compares to the AI baselines!
 
 ### Hackathon
 During a Microsoft hackathon July 27-29, 2020, several people completed 30 user 
-[study puzzles](/problems/README.md#study). We also had tons of fun making the puzzles in 
+[study puzzles](/puzzles/README.md#study). We also had tons of fun making the puzzles in 
 [Hackathon_puzzles.ipynb](/notebooks/Hackathon_puzzles.ipynb). These are of a somewhat
 different flavor as they are more often `hacks` like 
 ```python

templates/ICPC.py generators/ICPC.pytemplates/ICPC.py renamed to generators/ICPC.py

@@ -2,29 +2,31 @@
 Problems inspired by the [International Collegiate Programming Contest](https://icpc.global) (ICPC).
 """
 
-from problems import Problem
+from puzzle_generator import PuzzleGenerator
 from typing import List
 
 
-# Hint: subclass Problem.Debug for quick testing. Run make_dataset.py to make the dataset
-# See https://github.com/microsoft/PythonProgrammingPuzzles/wiki/How-to-add-a-puzzle for more info
+# See https://github.com/microsoft/PythonProgrammingPuzzles/wiki/How-to-add-a-puzzle to learn about adding puzzles
 
 
-class BiPermutations(Problem):
-    """There are two rows of objects. Given the length-n integer arrays of prices and heights of objects in each
-    row, find a permutation of both rows so that the permuted prices are non-decreasing in each row and
-    so that the first row is taller than the second row.
-
+class BiPermutations(PuzzleGenerator):
+    """
     Inspired by
     [ICPC 2019 Problem A: Azulejos](https://icpc.global/newcms/worldfinals/problems/2019%20ACM-ICPC%20World%20Finals/icpc2019.pdf)
-    which is 2,287 characters."""
+    which is 2,287 characters.
+    """
 
     @staticmethod
     def sat(perms: List[List[int]],
             prices0=[7, 7, 9, 5, 3, 7, 1, 2],
             prices1=[5, 5, 5, 4, 2, 5, 1, 1],
             heights0=[2, 4, 9, 3, 8, 5, 5, 4],
             heights1=[1, 3, 8, 1, 5, 4, 4, 2]):
+        """
+        There are two rows of objects. Given the length-n integer arrays of prices and heights of objects in each
+        row, find a permutation of both rows so that the permuted prices are non-decreasing in each row and
+        so that the first row is taller than the second row.
+        """
         n = len(prices0)
         perm0, perm1 = perms
         assert sorted(perm0) == sorted(perm1) == list(range(n)), "Solution must be two permutations"
@@ -74,20 +76,8 @@ def gen_random(self):
         self.add(dict(prices0=prices0, heights0=heights0, prices1=prices1, heights1=heights1))
 
 
-class OptimalBridges(Problem):
+class OptimalBridges(PuzzleGenerator):
     """
-    You are to choose locations for bridge bases from among a given set of mountain peaks located at
-    `xs, ys`, where `xs` and `ys` are lists of n integers of the same length. Your answer should be a sorted
-    list of indices starting at 0 and ending at n-1. The goal is to minimize building costs such that the bridges
-    are feasible. The bridges are all semicircles placed on top of the pillars. The feasibility constraints are that:
-    * The bridges may not extend above a given height `H`. Mathematically, if the distance between the two xs
-    of adjacent pillars is d, then the semicircle will have radius `d/2` and therefore the heights of the
-    selected mountain peaks must both be at most `H - d/2`.
-    *  The bridges must clear all the mountain peaks, which means that the semicircle must lie above the tops of the
-    peak. See the code for how this is determined mathematically.
-    * The total cost of all the bridges must be at most `thresh`, where the cost is parameter alpha * (the sum of
-    all pillar heights) + beta * (the sum of the squared diameters)
-
     Inspired by
     [ICPC 2019 Problem B: Bridges](https://icpc.global/newcms/worldfinals/problems/2019%20ACM-ICPC%20World%20Finals/icpc2019.pdf)
     which is 3,003 characters.
@@ -101,6 +91,19 @@ def sat(indices: List[int],
             xs=[0, 10, 20, 30, 50, 80, 100, 120, 160, 190, 200],
             ys=[0, 30, 10, 30, 50, 40, 10, 20, 20, 55, 10],
             thresh=26020):
+        """
+        You are to choose locations for bridge bases from among a given set of mountain peaks located at
+        `xs, ys`, where `xs` and `ys` are lists of n integers of the same length. Your answer should be a sorted
+        list of indices starting at 0 and ending at n-1. The goal is to minimize building costs such that the bridges
+        are feasible. The bridges are all semicircles placed on top of the pillars. The feasibility constraints are that:
+        * The bridges may not extend above a given height `H`. Mathematically, if the distance between the two xs
+        of adjacent pillars is d, then the semicircle will have radius `d/2` and therefore the heights of the
+        selected mountain peaks must both be at most `H - d/2`.
+        *  The bridges must clear all the mountain peaks, which means that the semicircle must lie above the tops of the
+        peak. See the code for how this is determined mathematically.
+        * The total cost of all the bridges must be at most `thresh`, where the cost is parameter alpha * (the sum of
+        all pillar heights) + beta * (the sum of the squared diameters)
+        """
         assert sorted({0, len(xs) - 1, *indices}) == indices, f"Ans. should be sorted list [0, ..., {len(xs) - 1}]"
         cost = alpha * (H - ys[0])
         for i, j in zip(indices, indices[1:]):
@@ -183,26 +186,30 @@ def gen_random(self):
         self.add(dict(H=H, alpha=alpha, beta=beta, xs=xs, ys=ys, thresh=thresh))
 
 
-class CheckersPosition(Problem):
-    """You are given a partial transcript a checkers game. Find an initial position such that the transcript
-    would be a legal set of moves. The board positions are [x, y] pairs with 0 <= x, y < 8 and x + y even.
-    There are two players which we call -1 and 1 for convenience, and player 1 must move first in transcript.
-    The initial position is represented as a list [x, y, piece] where piece means:
-    * 0 is empty square
-    * 1 or -1 is piece that moves only in the y = 1 or y = -1 dir, respectively
-    * 2 or -2 is king for player 1 or player 2 respectively
-
-    Additional rules:
-    * You must jump if you can, and you must continue jumping until one can't any longer.
-    * You cannot start the position with any non-kings on your last rank.
-    * Promotion happens after the turn ends
-
+class CheckersPosition(PuzzleGenerator):
+    """
     Inspired by
     [ICPC 2019 Problem C: Checks Post Facto](https://icpc.global/newcms/worldfinals/problems/2019%20ACM-ICPC%20World%20Finals/icpc2019.pdf)
+
+    Nobody solved this problem during the competition -- it is pretty difficult!
     """
 
     @staticmethod
     def sat(position: List[List[int]], transcript=[[[3, 3], [5, 5], [3, 7]], [[5, 3], [6, 4]]]):
+        """
+        You are given a partial transcript a checkers game. Find an initial position such that the transcript
+        would be a legal set of moves. The board positions are [x, y] pairs with 0 <= x, y < 8 and x + y even.
+        There are two players which we call -1 and 1 for convenience, and player 1 must move first in transcript.
+        The initial position is represented as a list [x, y, piece] where piece means:
+        * 0 is empty square
+        * 1 or -1 is piece that moves only in the y = 1 or y = -1 dir, respectively
+        * 2 or -2 is king for player 1 or player 2 respectively
+
+        Additional rules:
+        * You must jump if you can, and you must continue jumping until one can't any longer.
+        * You cannot start the position with any non-kings on your last rank.
+        * Promotion happens after the turn ends
+        """
         board = {(x, y): 0 for x in range(8) for y in range(8) if (x + y) % 2 == 0}  # empty board, 0 = empty
         for x, y, p in position:
             assert -2 <= p <= 2 and board[x, y] == 0  # -1, 1 is regular piece, -2, 2 is king
@@ -427,22 +434,23 @@ def make_random_move(sign):  # returns True if move was made else False if no le
         return [[list(a) for a in move] for move in transcript]  # convert to lists
 
 
-class MatchingMarkers(Problem):
+class MatchingMarkers(PuzzleGenerator):
     """
-    The input is a string of start and end markers "aaBAcGeg" where upper-case characters indicate start markers
-    and lower-case characters indicate ending markers. The string indicates a ring (joined at the ends) and the goal is
-    to find a location to split the ring so that there are a maximal number of matched start/end chars where a character
-    (like "a"/"A") is matched if starting at the split and going around the ring, the start-end pairs form a valid
-    nesting like nested parentheses.
-
-    This is trivial in quadratic time, but the challenge is to solve it quickly (i.e., linear time).
-
     Inspired by
     [ICPC 2019 Problem D: Circular DNA](https://icpc.global/newcms/worldfinals/problems/2019%20ACM-ICPC%20World%20Finals/icpc2019.pdf)
+
+    This is trivial in quadratic time, but the challenge is to solve it quickly (i.e., linear time).
     """
 
     @staticmethod
     def sat(cut_position: int, ring="yRrsmOkLCHSDJywpVDEDsjgCwSUmtvHMefxxPFdmBIpM", lower=5):
+        """
+        The input is a string of start and end markers "aaBAcGeg" where upper-case characters indicate start markers
+        and lower-case characters indicate ending markers. The string indicates a ring (joined at the ends) and the goal is
+        to find a location to split the ring so that there are a maximal number of matched start/end chars where a character
+        (like "a"/"A") is matched if starting at the split and going around the ring, the start-end pairs form a valid
+        nesting like nested parentheses. Can you solve it in linear time?
+        """
         line = ring[cut_position:] + ring[:cut_position]
         matches = {c: 0 for c in line.lower()}
         for c in line:
@@ -494,4 +502,4 @@ def gen_random(self):
 
 
 if __name__ == "__main__":
-    Problem.debug_problems()
+    PuzzleGenerator.debug_problems()