start of fitness function, coming back once I know more

Fix path for regression problem

.gitignore

@@ -23,6 +23,7 @@ notes
 /temp
 *~
 q
+scratch.clj
 
 # Clojure Script
 .shadow-cljs/
@@ -33,3 +34,5 @@ node_modules/
 # https://github.com/thelmuth/program-synthesis-benchmark-datasets
 /data
 **/.DS_Store
+*.edn
+/.cpcache/

PSB2/directory/path/goes/here/datasets/fizz-buzz/fizz-buzz-edge.edn

@@ -0,0 +1,24 @@
+{:input1 1, :output1 "1"}
+{:input1 2, :output1 "2"}
+{:input1 3, :output1 "Fizz"}
+{:input1 4, :output1 "4"}
+{:input1 5, :output1 "Buzz"}
+{:input1 6, :output1 "Fizz"}
+{:input1 7, :output1 "7"}
+{:input1 8, :output1 "8"}
+{:input1 9, :output1 "Fizz"}
+{:input1 10, :output1 "Buzz"}
+{:input1 11, :output1 "11"}
+{:input1 12, :output1 "Fizz"}
+{:input1 13, :output1 "13"}
+{:input1 14, :output1 "14"}
+{:input1 15, :output1 "FizzBuzz"}
+{:input1 16, :output1 "16"}
+{:input1 17, :output1 "17"}
+{:input1 18, :output1 "Fizz"}
+{:input1 19, :output1 "19"}
+{:input1 20, :output1 "Buzz"}
+{:input1 49995, :output1 "FizzBuzz"}
+{:input1 49998, :output1 "Fizz"}
+{:input1 49999, :output1 "49999"}
+{:input1 50000, :output1 "Buzz"}

README.md

@@ -2,35 +2,86 @@
 
 Yet another Push-based genetic programming system in Clojure.
 
+Propeller is a component of several active research projects and it is subject to change as a part of these activities.
+
+See the git commit comments for guidance with respect to recent changes.
+
+Some documentation is available at [https://lspector.github.io/propeller/](https://lspector.github.io/propeller/), but this should be evaluated in the context of the commit messages and current source code.
+
+## About this fork
+
+This is mainly for personal use to develop inside of Nix environment. Check out the original on package on [github](https://github.com/lspector/propeller).
+
+### How to jack into the project
+
+Once you've ran `nix develop` and vscodium has appeared, click the REPL
+button in the bottom left of the screen. Click 
+`Start your project with a REPL and connect (a.k.a Jack-in)`, select
+`Leiningen`, and don't select a profile, instead click `OK`. A window should appear to the right with Calva's REPL.
+
+Head over to `src/propeller/session.cljc` as described below and click
+`ctrl + alt + c` then `Enter` to source the session file.
+
 ## Usage
 
-If you have installed [leiningen](https://leiningen.org), which is a tool
-for running Clojure programs, then you can run Propeller on a genetic
-programming problem that is defined within this project from the command
-line with the command `lein run -m <namespace>`, replacing `<namespace>` 
+If you are working in a Clojure IDE with an integrated REPL, the first
+thing you may want to do is to open `src/propeller/session.cljc` and 
+evaluate the namespace declaration and the commented-out expressions 
+therein. These demonstrate core components of Propeller including
+complete genetic programming runs. When conducting complete genetic
+programming runs this way (using `gp/gp`), depending on your IDE you 
+may need to explicitly open and load the problem file before evaluating 
+the calls to `require` and `gp/gp`.
+
+To run Propeller from the command line, on a genetic programming problem 
+that is defined within this project, you will probably want to use either
+the Clojure [CLI tools](https://clojure.org/guides/deps_and_cli) or 
+[leiningen](https://leiningen.org). In the examples below, the leiningen
+and CLI commands are identical except that the former begin with
+`lein run -m`, while the latter begin with `clj -M -m`.
+
+To start a run use `clj -M -m <namespace>` or 
+`lein run -m <namespace>`, replacing `<namespace>` 
 with the actual namespace that you will find at the top of the problem file. 
 
-For example, you can run the simple-regression genetic programming problem with:
+For example, you can run the integer-regression genetic programming problem with:
 
 ```
-lein run -m propeller.problems.simple-regression
+clj -M -m propeller.problems.regression.integer-regression
+```
+or 
+
+```
+lein run -m propeller.problems.regression.integer-regression
 ```
 
 Additional command-line arguments may
 be provided to override the default key/value pairs specified in the 
 problem file, for example:
 
+```
+clj -M -m propeller.problems.regression.integer-regression :population-size 100
+```
+
+or
 
 ```
-lein run -m propeller.problems.simple-regression :population-size 100
+lein run -m propeller.problems.regression.integer-regression :population-size 100
 ```
 
 On Unix operating systems, including MacOS, you can use something
 like the following to send output both to the terminal
 and to a text file (called `outfile` in this example):
 
+
 ```
-lein run -m propeller.problems.simple-regression | tee outfile
+clj -M -m propeller.problems.regression.integer-regression | tee outfile
+```
+
+or
+
+```
+lein run -m propeller.problems.regression.integer-regression | tee outfile
 ```
 
 If you want to provide command line arguments that include
@@ -41,24 +92,22 @@ value for the `:variation` argument, which is a clojure map
 containing curly brackets that may confuse your shell:
 
 ```
-lein run -m propeller.problems.simple-regression :variation "{:umad 1.0}"
+clj -M -m propeller.problems.regression.integer-regression :variation "{:umad 1.0}"
 ```
 
-To run a genetic programming problem from a REPL, start
-your REPL for the project (e.g. with `lein repl` at the
-command line when in the project directory, or through your
-IDE) and then do something like the following (which in
-this case runs the simple-regression problem with 
-`:population-size` 100):
+or
 
 ```
-(require 'propeller.problems.simple-regression)
-(in-ns 'propeller.problems.simple-regression)
-(-main :population-size 100 :variation {:umad 1.0})
+lein run -m propeller.problems.regression.integer-regression :variation "{:umad 1.0}"
 ```
 
-If you want to run the problem with the default parameters,
-then you should call `-main` without arguments, as `(-main)`.
+For many genetic operator hyperparameters, collections may be provided in place of single values. When this is done, a random element of the collection will be chosen (with each being equally likely) each time the operator is used. When specied at the command line, these collections will also have to be quoted, for example with `:umad-rate "[0.01 0.05 0.1]"` to mean that UMAD rates of 0.01, 0.05, and 0.1 can be used.
+
+By default, Propeller will conduct many processes concurrently on multiple 
+cores using threads. If you  want to disable this behavior (for example, during 
+debugging) then provide the argument `:single-thread-mode` with the value `true`.
+Threads are not available in Javascript, so no processes are run concurrnetly
+when Propeller is run in Clojurescript.
 
 
 ## CLJS Usage
@@ -94,8 +143,9 @@ Calling `(-main)` will run the default genetic programming problem.
 
 ## Description
 
-Propel is an implementation of the Push programming 
-language and the PushGP genetic programming system in Clojure.
+Propeller is an implementation of the Push programming 
+language and the PushGP genetic programming system in Clojure, based
+on Tom Helmuth's little PushGP implementation [propel](https://github.com/thelmuth/propel).
 
 For more information on Push and PushGP see 
 [http://pushlanguage.org](http://pushlanguage.org).

deps.edn

@@ -0,0 +1,14 @@
+{:paths ["src" "target/classes"],
+ :deps
+ {org.clojure/clojure #:mvn{:version "1.10.0"},
+  org.clojure/clojurescript #:mvn{:version "1.9.946"},
+  org.clojure/test.check #:mvn{:version "1.1.0"},
+  net.clojars.schneau/psb2 #:mvn{:version "1.1.1"}
+  org.clojure/data.csv #:mvn{:version "1.0.1"}},
+ :mvn/repos {}
+ :codox {:extra-deps {codox/codox {:mvn/version "0.10.8"}}
+         :exec-fn codox.main/generate-docs
+         :exec-args {:doc-paths ["src/docs_src"]
+                     :output-path "docs"
+                     :metadata {:doc "FIXME: write docs" :doc/format :markdown}}
+         }}

doc/intro.md

@@ -1,32 +0,0 @@
-# Introduction to Propeller
-
-TODO: write [great documentation](http://jacobian.org/writing/what-to-write/)
-
-
-
-
-# Simplification
-
-To use Propeller's auto-simplification system, simply include the following four command line arguments when running a problem:
-
-```clojure
-:simplification? true
-```
-Toggle auto-simplification
-```clojure
-:simplification-k 4
-``` 
-This is the upper bound for elements deleted from the plushy every step. Every step, a number in $[1, k]$ of elements is deleted from the plushy representation of the solution.
-```clojure
-:simplification-steps 1000 
-```
-Number of simplification steps to perform
-```clojure
-:simplification-verbose? true 
-```
-whether or not to output simplification info into the output of the evolutionary run.
-The output with verbose adds the following lines to the output:
-```clojure
-{:start-plushy-length 42, :k 4}
-{:final-plushy-length 13, :final-plushy (:in1 :in1 :integer_quot :in1 :in1 :exec_dup :in1 :integer_mult close :exec_dup :integer_add 1 :integer_add)}
-```

docs/css/default.css

@@ -0,0 +1,551 @@
+body {
+    font-family: Helvetica, Arial, sans-serif;
+    font-size: 15px;
+}
+
+pre, code {
+    font-family: Monaco, DejaVu Sans Mono, Consolas, monospace;
+    font-size: 9pt;
+    margin: 15px 0;
+}
+
+h1 {
+    font-weight: normal;
+    font-size: 29px;
+    margin: 10px 0 2px 0;
+    padding: 0;
+}
+
+h2 {
+    font-weight: normal;
+    font-size: 25px;
+}
+
+h5.license {
+    margin: 9px 0 22px 0;
+    color: #555;
+    font-weight: normal;
+    font-size: 12px;
+    font-style: italic;
+}
+
+.document h1, .namespace-index h1 {
+    font-size: 32px;
+    margin-top: 12px;
+}
+
+#header, #content, .sidebar {
+    position: fixed;
+}
+
+#header {
+    top: 0;
+    left: 0;
+    right: 0;
+    height: 22px;
+    color: #f5f5f5;
+    padding: 5px 7px;
+}
+
+#content {
+    top: 32px;
+    right: 0;
+    bottom: 0;
+    overflow: auto;
+    background: #fff;
+    color: #333;
+    padding: 0 18px;
+}
+
+.sidebar {
+    position: fixed;
+    top: 32px;
+    bottom: 0;
+    overflow: auto;
+}
+
+.sidebar.primary {
+    background: #e2e2e2;
+    border-right: solid 1px #cccccc;
+    left: 0;
+    width: 250px;
+}
+
+.sidebar.secondary {
+    background: #f2f2f2;
+    border-right: solid 1px #d7d7d7;
+    left: 251px;
+    width: 200px;
+}
+
+#content.namespace-index, #content.document {
+    left: 251px;
+}
+
+#content.namespace-docs {
+    left: 452px;
+}
+
+#content.document {
+    padding-bottom: 10%;
+}
+
+#header {
+    background: #3f3f3f;
+    box-shadow: 0 0 8px rgba(0, 0, 0, 0.4);
+    z-index: 100;
+}
+
+#header h1 {
+    margin: 0;
+    padding: 0;
+    font-size: 18px;
+    font-weight: lighter;
+    text-shadow: -1px -1px 0px #333;
+}
+
+#header h1 .project-version {
+    font-weight: normal;
+}
+
+.project-version {
+    padding-left: 0.15em;
+}
+
+#header a, .sidebar a {
+    display: block;
+    text-decoration: none;
+}
+
+#header a {
+    color: #f5f5f5;
+}
+
+.sidebar a {
+    color: #333;
+}
+
+#header h2 {
+    float: right;
+    font-size: 9pt;
+    font-weight: normal;
+    margin: 4px 3px;
+    padding: 0;
+    color: #bbb;
+}
+
+#header h2 a {
+    display: inline;
+}
+
+.sidebar h3 {
+    margin: 0;
+    padding: 10px 13px 0 13px;
+    font-size: 19px;
+    font-weight: lighter;
+}
+
+.sidebar h3 a {
+    color: #444;
+}
+
+.sidebar h3.no-link {
+    color: #636363;
+}
+
+.sidebar ul {
+    padding: 7px 0 6px 0;
+    margin: 0;
+}
+
+.sidebar ul.index-link {
+    padding-bottom: 4px;
+}
+
+.sidebar li {
+    display: block;
+    vertical-align: middle;
+}
+
+.sidebar li a, .sidebar li .no-link {
+    border-left: 3px solid transparent;
+    padding: 0 10px;
+    white-space: nowrap;
+}
+
+.sidebar li .no-link {
+    display: block;
+    color: #777;
+    font-style: italic;
+}
+
+.sidebar li .inner {
+    display: inline-block;
+    padding-top: 7px;
+    height: 24px;
+}
+
+.sidebar li a, .sidebar li .tree {
+    height: 31px;
+}
+
+.depth-1 .inner { padding-left: 2px; }
+.depth-2 .inner { padding-left: 6px; }
+.depth-3 .inner { padding-left: 20px; }
+.depth-4 .inner { padding-left: 34px; }
+.depth-5 .inner { padding-left: 48px; }
+.depth-6 .inner { padding-left: 62px; }
+
+.sidebar li .tree {
+    display: block;
+    float: left;
+    position: relative;
+    top: -10px;
+    margin: 0 4px 0 0;
+    padding: 0;
+}
+
+.sidebar li.depth-1 .tree {
+    display: none;
+}
+
+.sidebar li .tree .top, .sidebar li .tree .bottom {
+    display: block;
+    margin: 0;
+    padding: 0;
+    width: 7px;
+}
+
+.sidebar li .tree .top {
+    border-left: 1px solid #aaa;
+    border-bottom: 1px solid #aaa;
+    height: 19px;
+}
+
+.sidebar li .tree .bottom {
+    height: 22px;
+}
+
+.sidebar li.branch .tree .bottom {
+    border-left: 1px solid #aaa;
+}
+
+.sidebar.primary li.current a {
+    border-left: 3px solid #a33;
+    color: #a33;
+}
+
+.sidebar.secondary li.current a {
+    border-left: 3px solid #33a;
+    color: #33a;
+}
+
+.namespace-index h2 {
+    margin: 30px 0 0 0;
+}
+
+.namespace-index h3 {
+    font-size: 16px;
+    font-weight: bold;
+    margin-bottom: 0;
+}
+
+.namespace-index .topics {
+    padding-left: 30px;
+    margin: 11px 0 0 0;
+}
+
+.namespace-index .topics li {
+    padding: 5px 0;
+}
+
+.namespace-docs h3 {
+    font-size: 18px;
+    font-weight: bold;
+}
+
+.public h3 {
+    margin: 0;
+    float: left;
+}
+
+.usage {
+    clear: both;
+}
+
+.public {
+    margin: 0;
+    border-top: 1px solid #e0e0e0;
+    padding-top: 14px;
+    padding-bottom: 6px;
+}
+
+.public:last-child {
+    margin-bottom: 20%;
+}
+
+.members .public:last-child {
+    margin-bottom: 0;
+}
+
+.members {
+    margin: 15px 0;
+}
+
+.members h4 {
+    color: #555;
+    font-weight: normal;
+    font-variant: small-caps;
+    margin: 0 0 5px 0;
+}
+
+.members .inner {
+    padding-top: 5px;
+    padding-left: 12px;
+    margin-top: 2px;
+    margin-left: 7px;
+    border-left: 1px solid #bbb;
+}
+
+#content .members .inner h3 {
+    font-size: 12pt;
+}
+
+.members .public {
+    border-top: none;
+    margin-top: 0;
+    padding-top: 6px;
+    padding-bottom: 0;
+}
+
+.members .public:first-child {
+    padding-top: 0;
+}
+
+h4.type,
+h4.dynamic,
+h4.added,
+h4.deprecated {
+    float: left;
+    margin: 3px 10px 15px 0;
+    font-size: 15px;
+    font-weight: bold;
+    font-variant: small-caps;
+}
+
+.public h4.type,
+.public h4.dynamic,
+.public h4.added,
+.public h4.deprecated {
+    font-size: 13px;
+    font-weight: bold;
+    margin: 3px 0 0 10px;
+}
+
+.members h4.type,
+.members h4.added,
+.members h4.deprecated {
+    margin-top: 1px;
+}
+
+h4.type {
+    color: #717171;
+}
+
+h4.dynamic {
+    color: #9933aa;
+}
+
+h4.added {
+    color: #508820;
+}
+
+h4.deprecated {
+    color: #880000;
+}
+
+.namespace {
+    margin-bottom: 30px;
+}
+
+.namespace:last-child {
+    margin-bottom: 10%;
+}
+
+.index {
+    padding: 0;
+    font-size: 80%;
+    margin: 15px 0;
+    line-height: 16px;
+}
+
+.index * {
+    display: inline;
+}
+
+.index p {
+    padding-right: 3px;
+}
+
+.index li {
+    padding-right: 5px;
+}
+
+.index ul {
+    padding-left: 0;
+}
+
+.type-sig {
+    clear: both;
+    color: #088;
+}
+
+.type-sig pre {
+    padding-top: 10px;
+    margin: 0;
+}
+
+.usage code {
+    display: block;
+    color: #008;
+    margin: 2px 0;
+}
+
+.usage code:first-child {
+    padding-top: 10px;
+}
+
+p {
+    margin: 15px 0;
+}
+
+.public p:first-child, .public pre.plaintext {
+    margin-top: 12px;
+}
+
+.doc {
+    margin: 0 0 26px 0;
+    clear: both;
+}
+
+.public .doc {
+    margin: 0;
+}
+
+.namespace-index .doc {
+    margin-bottom: 20px;
+}
+
+.namespace-index .namespace .doc {
+    margin-bottom: 10px;
+}
+
+.markdown p, .markdown li, .markdown dt, .markdown dd, .markdown td {
+    line-height: 22px;
+}
+
+.markdown li {
+    padding: 2px 0;
+}
+
+.markdown h2 {
+    font-weight: normal;
+    font-size: 25px;
+    margin: 30px 0 10px 0;
+}
+
+.markdown h3 {
+    font-weight: normal;
+    font-size: 20px;
+    margin: 30px 0 0 0;
+}
+
+.markdown h4 {
+    font-size: 15px;
+    margin: 22px 0 -4px 0;
+}
+
+.doc, .public, .namespace .index {
+    max-width: 680px;
+    overflow-x: visible;
+}
+
+.markdown pre > code {
+    display: block;
+    padding: 10px;
+}
+
+.markdown pre > code, .src-link a {
+    border: 1px solid #e4e4e4;
+    border-radius: 2px;
+}
+
+.markdown code:not(.hljs), .src-link a {
+    background: #f6f6f6;
+}
+
+pre.deps {
+    display: inline-block;
+    margin: 0 10px;
+    border: 1px solid #e4e4e4;
+    border-radius: 2px;
+    padding: 10px;
+    background-color: #f6f6f6;
+}
+
+.markdown hr {
+    border-style: solid;
+    border-top: none;
+    color: #ccc;
+}
+
+.doc ul, .doc ol {
+    padding-left: 30px;
+}
+
+.doc table {
+    border-collapse: collapse;
+    margin: 0 10px;
+}
+
+.doc table td, .doc table th {
+    border: 1px solid #dddddd;
+    padding: 4px 6px;
+}
+
+.doc table th {
+    background: #f2f2f2;
+}
+
+.doc dl {
+    margin: 0 10px 20px 10px;
+}
+
+.doc dl dt {
+    font-weight: bold;
+    margin: 0;
+    padding: 3px 0;
+    border-bottom: 1px solid #ddd;
+}
+
+.doc dl dd {
+    padding: 5px 0;
+    margin: 0 0 5px 10px;
+}
+
+.doc abbr {
+    border-bottom: 1px dotted #333;
+    font-variant: none;
+    cursor: help;
+}
+
+.src-link {
+    margin-bottom: 15px;
+}
+
+.src-link a {
+    font-size: 70%;
+    padding: 1px 4px;
+    text-decoration: none;
+    color: #5555bb;
+}

docs/css/highlight.css

@@ -0,0 +1,97 @@
+/*
+github.com style (c) Vasily Polovnyov <vast@whiteants.net>
+*/
+
+.hljs {
+  display: block;
+  overflow-x: auto;
+  padding: 0.5em;
+  color: #333;
+  background: #f8f8f8;
+}
+
+.hljs-comment,
+.hljs-quote {
+  color: #998;
+  font-style: italic;
+}
+
+.hljs-keyword,
+.hljs-selector-tag,
+.hljs-subst {
+  color: #333;
+  font-weight: bold;
+}
+
+.hljs-number,
+.hljs-literal,
+.hljs-variable,
+.hljs-template-variable,
+.hljs-tag .hljs-attr {
+  color: #008080;
+}
+
+.hljs-string,
+.hljs-doctag {
+  color: #d14;
+}
+
+.hljs-title,
+.hljs-section,
+.hljs-selector-id {
+  color: #900;
+  font-weight: bold;
+}
+
+.hljs-subst {
+  font-weight: normal;
+}
+
+.hljs-type,
+.hljs-class .hljs-title {
+  color: #458;
+  font-weight: bold;
+}
+
+.hljs-tag,
+.hljs-name,
+.hljs-attribute {
+  color: #000080;
+  font-weight: normal;
+}
+
+.hljs-regexp,
+.hljs-link {
+  color: #009926;
+}
+
+.hljs-symbol,
+.hljs-bullet {
+  color: #990073;
+}
+
+.hljs-built_in,
+.hljs-builtin-name {
+  color: #0086b3;
+}
+
+.hljs-meta {
+  color: #999;
+  font-weight: bold;
+}
+
+.hljs-deletion {
+  background: #fdd;
+}
+
+.hljs-addition {
+  background: #dfd;
+}
+
+.hljs-emphasis {
+  font-style: italic;
+}
+
+.hljs-strong {
+  font-weight: bold;
+}

docs/js/page_effects.js

@@ -0,0 +1,112 @@
+function visibleInParent(element) {
+    var position = $(element).position().top
+    return position > -50 && position < ($(element).offsetParent().height() - 50)
+}
+
+function hasFragment(link, fragment) {
+    return $(link).attr("href").indexOf("#" + fragment) != -1
+}
+
+function findLinkByFragment(elements, fragment) {
+    return $(elements).filter(function(i, e) { return hasFragment(e, fragment)}).first()
+}
+
+function scrollToCurrentVarLink(elements) {
+    var elements = $(elements);
+    var parent   = elements.offsetParent();
+
+    if (elements.length == 0) return;
+
+    var top    = elements.first().position().top;
+    var bottom = elements.last().position().top + elements.last().height();
+
+    if (top >= 0 && bottom <= parent.height()) return;
+
+    if (top < 0) {
+        parent.scrollTop(parent.scrollTop() + top);
+    }
+    else if (bottom > parent.height()) {
+        parent.scrollTop(parent.scrollTop() + bottom - parent.height());
+    }
+}
+
+function setCurrentVarLink() {
+    $('.secondary a').parent().removeClass('current')
+    $('.anchor').
+        filter(function(index) { return visibleInParent(this) }).
+        each(function(index, element) {
+            findLinkByFragment(".secondary a", element.id).
+                parent().
+                addClass('current')
+        });
+    scrollToCurrentVarLink('.secondary .current');
+}
+
+var hasStorage = (function() { try { return localStorage.getItem } catch(e) {} }())
+
+function scrollPositionId(element) {
+    var directory = window.location.href.replace(/[^\/]+\.html$/, '')
+    return 'scroll::' + $(element).attr('id') + '::' + directory
+}
+
+function storeScrollPosition(element) {
+    if (!hasStorage) return;
+    localStorage.setItem(scrollPositionId(element) + "::x", $(element).scrollLeft())
+    localStorage.setItem(scrollPositionId(element) + "::y", $(element).scrollTop())
+}
+
+function recallScrollPosition(element) {
+    if (!hasStorage) return;
+    $(element).scrollLeft(localStorage.getItem(scrollPositionId(element) + "::x"))
+    $(element).scrollTop(localStorage.getItem(scrollPositionId(element) + "::y"))
+}
+
+function persistScrollPosition(element) {
+    recallScrollPosition(element)
+    $(element).scroll(function() { storeScrollPosition(element) })
+}
+
+function sidebarContentWidth(element) {
+    var widths = $(element).find('.inner').map(function() { return $(this).innerWidth() })
+    return Math.max.apply(Math, widths)
+}
+
+function calculateSize(width, snap, margin, minimum) {
+    if (width == 0) {
+        return 0
+    }
+    else {
+        return Math.max(minimum, (Math.ceil(width / snap) * snap) + (margin * 2))
+    }
+}
+
+function resizeSidebars() {
+    var primaryWidth   = sidebarContentWidth('.primary')
+    var secondaryWidth = 0
+
+    if ($('.secondary').length != 0) {
+        secondaryWidth = sidebarContentWidth('.secondary')
+    }
+
+    // snap to grid
+    primaryWidth   = calculateSize(primaryWidth, 32, 13, 160)
+    secondaryWidth = calculateSize(secondaryWidth, 32, 13, 160)
+
+    $('.primary').css('width', primaryWidth)
+    $('.secondary').css('width', secondaryWidth).css('left', primaryWidth + 1)
+
+    if (secondaryWidth > 0) {
+        $('#content').css('left', primaryWidth + secondaryWidth + 2)
+    }
+    else {
+        $('#content').css('left', primaryWidth + 1)
+    }
+}
+
+$(window).ready(resizeSidebars)
+$(window).ready(setCurrentVarLink)
+$(window).ready(function() { persistScrollPosition('.primary')})
+$(window).ready(function() {
+    $('#content').scroll(setCurrentVarLink)
+    $(window).resize(setCurrentVarLink)
+})

flake.lock

@@ -0,0 +1,27 @@
+{
+  "nodes": {
+    "nixpkgs": {
+      "locked": {
+        "lastModified": 1741600792,
+        "narHash": "sha256-yfDy6chHcM7pXpMF4wycuuV+ILSTG486Z/vLx/Bdi6Y=",
+        "owner": "nixos",
+        "repo": "nixpkgs",
+        "rev": "ebe2788eafd539477f83775ef93c3c7e244421d3",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nixos",
+        "ref": "nixos-24.11",
+        "repo": "nixpkgs",
+        "type": "github"
+      }
+    },
+    "root": {
+      "inputs": {
+        "nixpkgs": "nixpkgs"
+      }
+    }
+  },
+  "root": "root",
+  "version": 7
+}

flake.nix

@@ -0,0 +1,42 @@
+{
+  description = "A flake for developing in a clojure environment with vscode";
+
+  inputs = {
+    nixpkgs.url = "github:nixos/nixpkgs/nixos-24.11";
+  };
+
+  outputs = { nixpkgs, ... }:
+  let system = "x86_64-linux"; in
+  {
+    devShells."${system}".default =
+    let
+      pkgs = nixpkgs.legacyPackages."${system}";
+    in
+      pkgs.mkShellNoCC {
+        buildInputs = [ pkgs.bashInteractive ];
+        packages = with pkgs; [
+          clojure
+          clojure-lsp
+          leiningen
+          nodejs
+          jdk8
+          (vscode-with-extensions.override {
+            vscode = vscodium;
+            vscodeExtensions = pkgs.vscode-utils.extensionsFromVscodeMarketplace [
+              {
+                name = "calva";
+                publisher = "betterthantomorrow";
+                version = "2.0.490";
+                sha256 = "sha256-PCEzSWahrTHXeGFrFShvbXrnoq3AtuVkoohKLxBGDRA=";
+              }
+            ];
+          })
+        ];
+        shellHook = ''
+          export SHELL=${pkgs.lib.getExe pkgs.bashInteractive}
+          echo "starting codium"
+          codium .
+        '';
+      };
+  };
+}

package-lock.json

@@ -907,9 +907,9 @@
       "dev": true
     },
     "node_modules/qs": {
-      "version": "6.11.0",
-      "resolved": "https://registry.npmjs.org/qs/-/qs-6.11.0.tgz",
-      "integrity": "sha512-MvjoMCJwEarSbUYk5O+nmoSzSutSsTwF85zcHPQ9OrlFoZOYIjaqBAJIqIXjptyD5vThxGq52Xu/MaJzRkIk4Q==",
+      "version": "6.10.1",
+      "resolved": "https://registry.npmjs.org/qs/-/qs-6.10.1.tgz",
+      "integrity": "sha512-M528Hph6wsSVOBiYUnGf+K/7w0hNshs/duGsNXPUCLH5XAqjEtiPGwNONLV0tBH8NoGb0mvD5JubnUTrujKDTg==",
       "dev": true,
       "dependencies": {
         "side-channel": "^1.0.4"
@@ -2096,9 +2096,9 @@
       "dev": true
     },
     "qs": {
-      "version": "6.11.0",
-      "resolved": "https://registry.npmjs.org/qs/-/qs-6.11.0.tgz",
-      "integrity": "sha512-MvjoMCJwEarSbUYk5O+nmoSzSutSsTwF85zcHPQ9OrlFoZOYIjaqBAJIqIXjptyD5vThxGq52Xu/MaJzRkIk4Q==",
+      "version": "6.10.1",
+      "resolved": "https://registry.npmjs.org/qs/-/qs-6.10.1.tgz",
+      "integrity": "sha512-M528Hph6wsSVOBiYUnGf+K/7w0hNshs/duGsNXPUCLH5XAqjEtiPGwNONLV0tBH8NoGb0mvD5JubnUTrujKDTg==",
       "dev": true,
       "requires": {
         "side-channel": "^1.0.4"

project.clj

@@ -1,4 +1,4 @@
-(defproject net.clojars.lspector/propeller "0.3.0"
+(defproject net.clojars.lspector/propeller "0.3.2"
   :description "Yet another Push-based genetic programming system in Clojure."
   :url "https://github.com/lspector/propeller"
   :license {:name "EPL-2.0 OR GPL-2.0-or-later WITH Classpath-exception-2.0"
@@ -6,8 +6,13 @@
   :dependencies [[org.clojure/clojure "1.10.0"]
                  [org.clojure/clojurescript "1.9.946"]
                  [org.clojure/test.check "1.1.0"]
-                 [net.clojars.schneau/psb2 "1.1.0"]]
+                 [net.clojars.schneau/psb2 "1.1.1"]
+                 [org.clojure/data.csv "1.0.1"]]
   :profiles {:profiling {:dependencies [[com.clojure-goes-fast/clj-async-profiler "0.5.1"]]}}
   :main ^:skip-aot propeller.core
   :repl-options {:init-ns propeller.core}
-  :jvm-opts ^:replace [])
+  :jvm-opts ^:replace []
+  :plugins [[lein-codox "0.10.8"]]
+  :codox {:output-path "docs"
+          :metadata {:doc "FIXME: write docs" :doc/format :markdown}
+          :doc-paths ["src/docs_src"]})

scripts/FunctionsToMD.py

@@ -0,0 +1,73 @@
+
+# Takes Push instructions defined through (def-instruction and
+# puts their documentation into a markdown file in docs_src
+
+import os
+from mdutils.mdutils import MdUtils
+mdFile = MdUtils(file_name='src/docs_src/Additional_Instructions')
+mdFile.new_header(level=1, title='Additional Instructions')
+os.chdir('..')
+
+instructionFiles = os.listdir('src/propeller/push/instructions')
+instructionFiles.remove('vector.cljc')
+instructionFiles.remove('polymorphic.cljc')
+print(instructionFiles)
+
+hasDefInstruction = False
+for file in instructionFiles:
+    mdFile.new_header(level=1, title=file)
+    try:
+        print(file)
+
+        # opening and reading the file
+        file_read = open('src/propeller/push/instructions/'+file, "r")
+
+        # set search text
+        text = "(def-instruction"
+
+        # reading file content line by line.
+        lines = file_read.readlines()
+
+        # looping through each line in the file
+        # if the line contains "\(def-instruction", go through lines above that line and add
+        # the Clojure comments to a list which is later written into markdown file.
+        for count, line in enumerate(lines):
+            new_list = []
+            # print(line)
+            # print(count)
+            if text in line:
+                hasDefInstruction = True
+                # print(line)
+                mdFile.new_header(level=2, title=lines[count+1].strip())
+                isComment = True
+                inc = 1
+                while isComment:
+                    if lines[count-inc].startswith(';;'):
+                        new_list.append(lines[count-inc].replace(';', '').strip())
+                        # print(lines[count-inc])
+                        inc = inc + 1
+                    else:
+                        isComment = False
+                        new_list.reverse()
+                        for comment in new_list:
+                            mdFile.write(comment+' ')
+                functionInfo = lines[count+1].strip() + lines[count-1].replace(';', '').strip()
+                # print(functionInfo)
+                new_list.append(functionInfo)
+        # closing file after reading
+        file_read.close()
+
+        # the input string doesn't
+        # found in the text file
+        if not hasDefInstruction:
+            print("\n\"" + text + "\" is not found in \"" + file + "\"!")
+            mdFile.new_paragraph('')
+        else:
+            print("There is"+text)
+
+    # entering except block
+    # if input file doesn't exist
+    except:
+        print("\nThe file doesn't exist!")
+mdFile.new_table_of_contents()
+mdFile.create_md_file()

scripts/GenerateDocs.sh

@@ -0,0 +1,6 @@
+#!/bin/sh
+
+pip install mdutils
+lein codox
+python3 FunctionsToMD.py
+python3 HTMLFix.py

scripts/HTMLFix.py

@@ -0,0 +1,27 @@
+
+# fixes ordered lists in codox-generated HTML for specific files
+
+import os
+
+os.chdir('..')
+htmlFiles = ['Adding_Genetic_Operators.html', 'Adding_Problem.html', 'Adding_Selection_Method.html']
+
+
+for file in htmlFiles:
+
+    with open('docs/'+file, 'r') as f:
+        OL = "ol>"
+        countOL = 0
+        newline = []
+        for line in f.readlines():
+            if OL in line:
+                countOL = countOL + 1
+                if countOL != 2 and countOL != 3 and countOL != 6 and countOL != 7:
+                    newline.append(line)
+            else:
+                newline.append(line)
+
+    with open('docs/'+file, 'w') as f:
+        for line in newline:
+            f.writelines(line)
+

src/docs_src/A_Guide_To_Propeller.md

@@ -0,0 +1,202 @@
+# A Guide to Propeller
+
+**Propeller**  is an implementation of the Push programming language and the PushGP genetic programming system in Clojure.
+
+For more information on Push and PushGP see http://pushlanguage.org.
+
+## Overview
+
+**Propeller** is a Push-based genetic programming system in Clojure.
+
+<!-- TOC -->
+* [A Guide to Propeller](#a-guide-to-propeller)
+  * [Overview](#overview)
+    * [What can you do with Propeller?](#what-can-you-do-with-propeller)
+  * [Installation](#installation)
+  * [How do I run Propeller on a problem?](#how-do-i-run-propeller-on-a-problem)
+    * [An Example](#an-example)
+    * [Can you use a REPL?](#can-you-use-a-repl)
+  * [Tutorials](#tutorials)
+  * [Contributing](#contributing)
+  * [License](#license)
+  * [Citation](#citation)
+  * [Contact](#contact)
+<!-- TOC -->
+
+### What can you do with Propeller?
+
+You can evolve a Push program to solve a problem. 
+You can also use the Push interpreter to evaluate Push programs in other projects, 
+for example in agent-based evolutionary simulations in which 
+agents are controlled by evolving Push programs.
+
+## Installation
+
+If you have installed [leiningen](https://leiningen.org), which is a tool
+for running Clojure programs, then you can run Propeller on a genetic
+programming problem that is defined within this project from the command
+line with the command `lein run -m <namespace>`, replacing `<namespace>`
+with the actual namespace that you will find at the top of the problem file.
+
+If you have installed [Clojure](https://clojure.org/guides/install_clojure#java), you can run Propeller on a genetic programming 
+problem with the command `clj -M -m <namespace>`, replacing `<namespace>` with 
+the actual namespace that you will find at the top of the problem file. 
+The examples below use leiningen, but you can replace `lein run -m` with `clj -M -m` to run the same problem.
+
+A specific example is provided later below.
+
+## How do I run Propeller on a problem?
+
+To run Propeller on a problem, you want to call the `-main` function in the problem file using leiningen. 
+The `-main` function will create a map of arguments from the input and run the main genetic programming loop.
+
+Below is the general format to run a problem through the command-line:
+
+```
+lein run -m [namespace of the problem file you want to test]
+```
+
+Additional command-line arguments may
+be provided to override the default key/value pairs specified in the
+problem file,
+
+```
+lein run -m [namespace of the problem file you want to test] [key and value] [key and value]...
+```
+
+The possible keys come from the table below:
+
+| Key                        | Description                                                                                                                                                                                                                                                                 |
+|----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `:instructions`            | List of possible Push instructions used to create a plushy                                                                                                                                                                                                                  |
+| `:error-function`          | The error function used to evaluate individuals, specified in the given problem's namespace                                                                                                                                                                                 |
+| `:training-data`           | Map of inputs and desired outputs used to evaluate individuals of the form: {:input1 first-input :input2 second-input ... :output1 first-output ...}                                                                                                                        |
+| `:testing-data`            | Map of inputs and desired outputs not in the training-data to test generalizability of a program that fits the `training-data`. The map is of the form: {:input1 first-input :input2 second-input ... :output1 first-output ...}                                            |
+| `:max-generations`         | Maximum number of generations                                                                                                                                                                                                                                               |
+| `:population-size`         | Size of population in a generation                                                                                                                                                                                                                                          |
+| `:max-initial-plushy-size` | Maximum number of Push instructions in the initial plushy                                                                                                                                                                                                                   |
+| `:step-limit`              | The maximum number of steps that a Push program will be executed by `interpret-program`                                                                                                                                                                                     |
+| `:parent-selection`        | Function from `propeller.selection` that determines method of parent selection method. Propeller includes `:tournament-selection`, `:lexicase-selection`, and `:epsilon-lexicase-selection`.                                                                                |
+| `:tournament-size`         | If using a tournament selection method, the number of individuals in each tournaments used to determine parents                                                                                                                                                             |
+| `:umad-rate`               | Rate (decimal between 0 and 1) of uniform mutation by addition and deletion (UMAD) genetic operator                                                                                                                                                                         |
+| `:variation`               | Map with genetic operators as keys and probabilities as values. For example, {:umad 0.3 :crossover 0.7}. This would mean that when the system needs to generate a child, it will use UMAD 30% of the time and crossover 70% of the time. The probabilities should sum to 1. |
+| `:elitism`                 | When true, will cause the individual with the lowest error in the population to survive, without variation, into the next generation.                                                                                                                                       |
+
+When you run a problem, you will get a report each generation with the following information:
+
+```
+ :generation            
+ :best-plushy 
+ :best-program          
+ :best-total-error      
+ :best-errors        
+ :best-behaviors        
+ :genotypic-diversity   
+ :behavioral-diversity  
+ :average-genome-length 
+ :average-total-error 
+
+```
+
+### An Example
+
+For example, you can run the simple-regression genetic programming problem with:
+
+```
+lein run -m propeller.problems.simple-regression
+```
+
+This will run simple-regression with the default set of arguments in the `simple-regression` problem file.
+
+```
+{:instructions             instructions
+:error-function           error-function
+:training-data            (:train train-and-test-data)
+:testing-data             (:test train-and-test-data)
+:max-generations          500
+:population-size          500
+:max-initial-plushy-size  100
+:step-limit               200
+:parent-selection         :lexicase
+:tournament-size          5
+:umad-rate                0.1
+:variation                {:umad 0.5 :crossover 0.5}
+:elitism                  false}
+```
+
+You can override the default key/value pairs with additional arguments. For example:
+
+```
+lein run -m propeller.problems.simple-regression :population-size 100
+```
+
+On Unix operating systems, including MacOS, you can use something
+like the following to send output both to the terminal
+and to a text file (called `outfile` in this example):
+
+```
+lein run -m propeller.problems.simple-regression | tee outfile
+```
+
+If you want to provide command line arguments that include
+characters that may be interpreted by your command line shell
+before they get to Clojure, then enclose those in double
+quotes, like in this example that provides a non-default
+value for the `:variation` argument, which is a clojure map
+containing curly brackets that may confuse your shell:
+
+```
+lein run -m propeller.problems.simple-regression :variation "{:umad 1.0}"
+```
+
+### Can you use a REPL?
+
+Yes!
+
+To run a genetic programming problem from a REPL, start
+your REPL for the project (e.g. with `lein repl` at the
+command line when in the project directory, or through your
+IDE) and then do something like the following (which in
+this case runs the simple-regression problem with
+`:population-size` 100):
+
+```
+(require 'propeller.problems.simple-regression)
+(in-ns 'propeller.problems.simple-regression)
+(-main :population-size 100 :variation {:umad 1.0})
+```
+
+If you want to run the problem with the default parameters,
+then you should call `-main` without arguments, as `(-main).
+
+## Tutorials
+
+- [Adding genetic operators](Adding_Genetic_Operators.md)
+- [Adding selection methods](Adding_Selection_Method.md)
+- [Adding a new problem](Adding_Problem.md)
+- [Generating Documentation](Generating_Documentation.md)
+
+## Contributing
+
+You can report a bug on the [GitHub issues page](https://github.com/lspector/propeller/issues).
+
+The best way to contribute to Propeller is to fork the [main GitHub repository](https://github.com/lspector/propeller) and submit a pull request.
+
+Propeller provides a way to automatically [generate documentation](Generating_Documentation.md) for any contributions
+you might make.
+
+## License
+
+**Eclipse Public License 2.0**
+
+This commercially-friendly copyleft license provides the ability to commercially license binaries; 
+a modern royalty-free patent license grant; and the ability for linked works to use other licenses, including commercial ones.
+
+## Citation
+
+We are in the process of creating a DOI, but in the meantime, 
+we ask that you cite the [link to the repository](https://github.com/lspector/propeller) if you use Propeller.
+
+## Contact
+
+To discuss Propeller, Push, and PushGP, you can join the [Push-Language Discourse](https://discourse.pushlanguage.org/).

src/docs_src/Adding_Genetic_Operators.md

@@ -0,0 +1,62 @@
+# Adding Genetic Operators
+
+In addition to the already-included genetic operators, you can add your own!
+
+## Variation Genetic Operators
+
+1. Go to `propeller.variation.cljc`
+2. Define a genetic operator function
+3. In `propeller.variation/new-individual`, add the new genetic operator in the `new-individual` function under the `case` call
+
+``` clojure
+(defn new-individual
+  "Returns a new individual produced by selection and variation of
+  individuals in the population."
+  [pop argmap]
+  ...
+     (case op
+       ...
+       
+       :new-genetic-operator
+       (-> (:plushy (selection/select-parent pop argmap))
+           (new-genetic-operator ))
+       ...
+       :else
+       (throw #?(:clj  (Exception. (str "No match in new-individual for " op))
+                 :cljs (js/Error
+                         (str "No match in new-individual for " op))))))})
+```
+
+4. When running a problem, specify the genetic operator in `:variation`.
+For example:
+```
+lein run -m propeller.problems.simple-regression :variation "{:new-genetic-operator 1.0}"
+
+```
+
+## Selection Genetic Operators
+
+1. Go to `propeller.selection.cljc`
+2. Define a genetic operator function
+3. In `propeller.selection.cljc`, add the new genetic operator in the `select-parent` function under the `case` call.
+
+```clojure
+(defn select-parent
+  "Selects a parent from the population using the specified method."
+  [pop argmap]
+  (case (:parent-selection argmap)
+    ...
+    :new-genetic-operator (:new-genetic-operator )
+    ...
+    ))
+```
+
+4. When running a problem, specify the selection method in `:parent-selection`
+
+For example:
+
+```
+lein run -m propeller.problems.simple-regression :parent-selection :new-genetic-operator
+
+```
+

src/docs_src/Adding_Problem.md

@@ -0,0 +1,121 @@
+# Adding a Problem
+
+In general, a problem file has 3 components: `train-and-test-data`, `instructions`, `error-function`, and `-main`.
+
+1. To add a new problem, you need training and test data. For Problem Synthesis Benchmark Problems (PSB2),
+you can fetch datasets using `psb2.core/fetch-examples`.
+
+```clojure
+(defn fetch-examples
+  "Fetches and returns training and test data from a PSB2 problem.
+   Returns a map of the form {:train training-examples :test testing-examples}
+   where training-examples and testing-examples are lists of training and test
+   data. The elements of these lists are maps of the form:
+   {:input1 first-input :input2 second-input ... :output1 first-output ...}
+   The training examples will include all hard-coded edge cases included in the suite,
+   along with enough random cases to include `n-train` cases.
+   Note that this function loads large datasets and can be slow, 30-120 seconds.
+   Parameters:
+     `datasets-directory` - Location of the PSB2 datasets as downloaded from https://zenodo.org/record/4678739
+     `problem-name` - Name of the PSB2 problem, lowercase and seperated by dashes.
+         - Ex: indices-of-substring
+     `n-train` - Number of training cases to return
+     `n-test` - Number of test cases to return"
+  [datasets-directory problem-name n-train n-test]
+
+```
+2. Define the possible Push instructions to be used to create plushys. It should be a non-lazy list of 
+instructions from `push/instructions`, input instructions, close, and constants (including functions that produce constants).
+3. Define an error function that will evaluate plushys and add `:behaviors parsed-outputs`, 
+   `:errors`, and `:total-error` to the individual
+4. Define the function `-main` with a map of default arguments.
+
+## Example of a Problem
+
+```clojure
+(ns propeller.problems.PSB2.solve-boolean
+  (:require [psb2.core :as psb2]
+            [propeller.genome :as genome]
+            [propeller.push.interpreter :as interpreter]
+            [propeller.utils :as utils]
+            [propeller.push.instructions :refer [get-stack-instructions]]
+            [propeller.push.state :as state]
+            [propeller.gp :as gp]
+            #?(:cljs [cljs.reader :refer [read-string]])))
+
+; ===========  PROBLEM DESCRIPTION  ================================
+; SOLVE BOOLEAN from PSB2
+; Given a string representing a Boolean
+; expression consisting of T, F, |, and &, evaluate it and return
+; the resulting Boolean.
+;
+; Source: https://arxiv.org/pdf/2106.06086.pdf
+; ==================================================================
+
+(def train-and-test-data (psb2/fetch-examples "data" "solve-boolean" 200 2000))
+
+(def instructions
+  (utils/not-lazy
+    (concat
+      ;;; stack-specific instructions
+      (get-stack-instructions #{:exec :integer :boolean :char :string :print})
+      ;;; input instructions
+      (list :in1)
+      ;;; close
+      (list 'close)
+      ;;; ERCs (constants)
+      (list true false \t \f \& \|))))
+
+(defn error-function
+  [argmap data individual]
+  (let [program (genome/plushy->push (:plushy individual) argmap)
+        inputs (map (fn [i] (get i :input1)) data)
+        correct-outputs (map (fn [i] (get i :output1)) data)
+        outputs (map (fn [input]
+                       (state/peek-stack
+                         (interpreter/interpret-program
+                           program
+                           (assoc state/empty-state :input {:in1 input})
+                           (:step-limit argmap))
+                         :boolean))
+                     inputs)
+        parsed-outputs (map (fn [output]
+                              (try (read-string output)
+                                   #?(:clj  (catch Exception e 1000.0)
+                                      :cljs (catch js/Error. e 1000.0))))
+                            outputs)
+        errors (map (fn [correct-output output]
+                      (if (= output :no-stack-item)
+                        10000
+                        (if (= correct-output output)
+                           0
+                           1)))
+                    correct-outputs
+                    parsed-outputs)]
+    (assoc individual
+      :behaviors parsed-outputs
+      :errors errors
+      :total-error #?(:clj  (apply +' errors)
+                      :cljs (apply + errors)))))
+
+(defn -main
+  "Runs propel-gp, giving it a map of arguments."
+  [& args]
+  (gp/gp
+    (merge
+      {:instructions            instructions
+       :error-function          error-function
+       :training-data           (:train train-and-test-data)
+       :testing-data            (:test train-and-test-data)
+       :max-generations         300
+       :population-size         1000
+       :max-initial-plushy-size 250
+       :step-limit              2000
+       :parent-selection        :lexicase
+       :tournament-size         5
+       :umad-rate               0.1
+       :variation               {:umad 1.0 :crossover 0.0}
+       :elitism                 false}
+      (apply hash-map (map #(if (string? %) (read-string %) %) args)))))
+
+```

src/docs_src/Adding_Selection_Method.md

@@ -0,0 +1,20 @@
+# Adding a Selection Method
+
+1. Define a selection method function in `propeller.selection` that selects an individual from the population
+2. Add the selection method in `propeller.selection/select-parent` under the `case` call:
+
+   ```clojure
+    (defn select-parent
+    "Selects a parent from the population using the specified method."
+    [pop argmap]
+    (case (:parent-selection argmap)
+    :new-selection-method (new-selection-method )))
+   ```
+
+3. When runnning a problem, specify the selection method in `:parent-selection`. 
+For example:
+    ```
+    lein run -m propeller.problems.simple-regression :parent-selection :new-selection-method"
+    ```
+
+