Added Hackathon_puzzles.ipynb!

Removed @register command using Problem.__subclasses__() from all templates
Changed the debug format to subclass Problem.Debug
Sped up generation so that it does not re-test solutions that were tested in the current json
Added a handful of puzzles
Added aka.ms link to the notebook on Binder

Author: akalai

.gitignore

@@ -24,6 +24,7 @@ pip-wheel-metadata/
 share/python-wheels/
 *.egg-info/
 .installed.cfg
+.idea
 *.egg
 MANIFEST
 

README.md

@@ -5,6 +5,21 @@ an AI's programming proficiency. We hope this dataset with **grow rapidly**, and
 terms of problem difficult, domain, 
 and algorithmic tools needed to solve the problems. Please [contribute](../../wiki/How-to-add-a-puzzle)! 
 
+To learn more about how well AI systems such as GPT-3 can solve these problems, read our paper:
+
+[Programming Puzzles](https://arxiv.org/abs/2106.05784). Tal Schuster, Ashwin Kalyan, Oleksandr Polozov, 
+Adam Tauman Kalai.
+```
+@misc{schuster2021programming,
+      title={Programming Puzzles}, 
+      author={Tal Schuster and Ashwin Kalyan and Oleksandr Polozov and Adam Tauman Kalai},
+      year={2021},
+      eprint={2106.05784},
+      archivePrefix={arXiv},      
+}
+```
+
+
 
 ## What is a python programming puzzle?
 
@@ -58,6 +73,42 @@ and [math problems](https://en.wikipedia.org/wiki/List_of_unsolved_problems_in_m
 * The website [codeforces.com](https://codeforces.com), a popular website for programming competition problems
 * Olympiad problems from the [International Collegiate Programming Contest](https://icpc.global) and [International Mathematical Olympiad](https://en.wikipedia.org/wiki/International_Mathematical_Olympiad).
 
+
+## Notebooks
+
+The [notebooks](/notebooks) subdirectory has some relevant notebooks. [Demo.ipynb](/notebooks/Demo.ipynb)
+has the 30 problems completed by our users in a user study. [Try the notebook at Binder](https://aka.ms/python_puzzles)
+and see how your programming compares to the AI baselines! 
+
+### Hackathon
+During a Microsoft hackathon July 27-29, 2020, several people completed 30 user 
+[study puzzles](/problems/README.md#study). We also had tons of fun making the puzzles in 
+[Hackathon_puzzles.ipynb](/notebooks/Hackathon_puzzles.ipynb). These are of a somewhat
+different flavor as they are more often `hacks` like 
+```python
+def f(x):
+    return x > x
+```
+where the type of `x` is clearly non-standard. The creators of these puzzles include github users: 
+    [Adam Tauman Kalai](akalai),
+    [Alec Helbling](helblazer811),
+    [Alexander Vorobev](OnFireDolphin),
+    [Alexander Wei](aw31),
+    [Alexey Romanov](jgc128),
+    [Keith Battaochi](kbattocchi),
+    [Maggie Hei](heimengqi),
+    [Misha Khodak](mkhodak),
+    [Monil Mehta](monilm2),
+    [Philip Rosenfield](philrosenfield),
+    [Qida Ma](JerryMa90),
+    [Raj Bhargava](rajbhargava),
+    [Rishi Jaiswal](nextquanta),
+    [Saikiran Mullaguri](sm947),
+    [Tal Schuster](TalSchuster), and
+    [Varsha Srinivasan](varsha2197). 
+You can try out the notebook at (link to be added).
+
+
 ## Highlights
 * Numerous trivial puzzles like reversing a list, useful for learning to program 
 * Classic puzzles like:

make_dataset.py

@@ -5,6 +5,7 @@
 import problems
 import utils
 import templates  # This loads all the problem templates
+import inspect
 
 TARGET_NUM_PER_PROBLEM = 1000
 
@@ -27,15 +28,24 @@
 
 def main(args):
     start_time = time.perf_counter()
-    templates = fnmatch.filter(problems.problem_registry.keys(), args.templates)
-    utils.info(f"Generating from templates: {templates}")
+    if problems.Problem.Debug.subclass_descendents(): # if there are debug problems, don't make the dataset
+        problems.Problem.debug_problems()
+        print("Didn't make dataset because there are `Problem.Debug` problems, remove the `.Debug` to proceed.")
+        return
+
+    all_probs = problems.Problem.subclass_descendents()
+
+    probs_by_template = utils.inv_dict({p: p.__module__.split(".")[-1] for p in all_probs})
+
+    used_templates = fnmatch.filter(probs_by_template, args.templates)
+    utils.info(f"Generating from templates: {used_templates}")
     problem_sets = []
-    for name in templates:  # order determined by import order in templates/__init__.py
-        entry = problems.problem_registry[name]
-        ps = problems.ProblemSet(entry["name"], entry["summary"])
-        for cls in entry["classes"]:
+    for name in used_templates:  # order determined by import order in templates/__init__.py
+        probs = probs_by_template[name]
+        ps = problems.ProblemSet(name, inspect.getmodule(probs[0]).__doc__)
+        for cls in probs:
             problem = cls()
-            problem.build(args.target_num_per_problem)
+            problem.build(args.target_num_per_problem, ps.get_already_tested())
             ps.add(problem)
         ps.save()
         problem_sets.append(ps)