Skip to content

Commit d409f13

Browse files
tekknolagitekknolagi
committed
Add a REPL webserver
This adds a server that acts as a function (exp, env) -> (result, env') meaning the server has to hold no state. The client maintains and transmits the state. This has some bugs right now, such as not being able to serialize recursive functions. Those will be fixed soon. --- This commit was comprised of the following commits: Add initial ScrapMonad Add a notion of relocations We'll need to link native functions, for example, if we take in a serialized environment from the outside. Add small server with JSON API and little HTML page Provide a networked REPL, perhaps as a demo. Add bdecode This is a little more stateful than bencode so add a class and call into the class from a `bdecode` function. Begin deserialization Use some Python nonsense to automatically add deserializer functions to a dict if Object subclasses have them. Add some more deserializers, including first recursive one This is a little painful and I wonder if there is a better way to do it. Some niceties Add deserialization for EnvObject Deserialize env and wrap it in EnvObject We need to pass around entire scrap objects. Persist env in client Serialize/deserialize functions, closures Next up: NativeFunctionRelocation. Type ScrapMonad.__init__ Drop the _ It's cleaner. Fix infinite recursion bug All Objects have deserialize (defined or inheritecd), but all of them *define* it. Check if they define it by looking in `__dict__`. Add support for deserializing NativeFunctionRelocation With this, the web REPL can ship an env back and forth to the idempotent eval server! Fix Python 3.8 Unwrap classmethod and store inner function. Kind of surprised mypy didn't catch this, since I was never storing functions before--always classmethod. Persist env in LocalStorage Persist history in LocalStorage Focus input field on page load Add TODO Throw some errors that do not normally raise Don't double fetch Add title and note Remove begin/end newlines Bound output window height and scroll to bottom Add doctype Embed empty favicon in page Save a request that will always 404. Add a little favicon Thanks, antifavicon. Set 1 hour cache control on eval requests As the server is stateless, we can have some intermediate server cache eval(env, exp) requests. If everyone loads the page and types `123`, for example, we only need receive that request once (per hour). It's not clear if fly.io (for example) does this middle cache thing automatically. Try building image in CI (#19) * Try building image in CI * wip * Add quick check post-build * Don't be interactive * Use tags from meta step to identify image Oops, remove target branch Remove excess env Scroll to bottom after loading history Add space in favicon Add button to clear LocalStorage Add TODO Add multiprocessing server Serialize/deserialize Apply Check in fly.toml Make web repl return result, not _ Report eval errors somewhat opaquely in console Add updateHistory function Use try/catch in client Display server error message in output Move all explicit exception handling into do_index Link to repl page Add charset and viewport Allow reuse of address This means we can C-c the server and re-start it on the same port immediately. Check in scrapscript.org style Adjust color and use <main> for padding Add fonts Render REPL with code/result a little better Bundle style.css with Docker image Fix input background Use prompt as label Remove TODO Don't highlight input box Deserialize bool and record Move html to separate file Expand input into textarea with Shift-Enter Add goatcounter Remove console.log Color textareas too Make the webserver fork Bring Chris's code up to date Print host in debug output
1 parent 05d2e00 commit d409f13

6 files changed

Lines changed: 611 additions & 2 deletions

File tree

‎Dockerfile‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,8 @@ RUN bin/ape.elf bin/python -m compileall Lib
1212
RUN mv Lib/__pycache__/scrapscript*.pyc Lib/scrapscript.pyc
1313
RUN rm Lib/scrapscript.py
1414
RUN cp bin/python bin/scrapscript.com
15+
COPY style.css Lib
16+
COPY repl.html Lib
1517
RUN sh bin/zip -A -r bin/scrapscript.com Lib .args
1618
RUN bin/ape.elf bin/assimilate bin/scrapscript.com
1719

‎build-com‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,14 +4,17 @@ PREV="$(pwd)"
44
DIR="$(mktemp -d)"
55
cp scrapscript.py "$DIR"
66
cp .args "$DIR"
7+
cp repl.html "$DIR"
8+
cp style.css "$DIR"
79
cd "$DIR"
810
wget https://cosmo.zip/pub/cosmos/bin/python
911
wget https://cosmo.zip/pub/cosmos/bin/zip
1012
chmod +x python
1113
chmod +x zip
1214
./python -m compileall scrapscript.py
1315
mkdir Lib
14-
cp __pycache__/scrapscript.cpython-311.pyc Lib/scrapscript.pyc
16+
cp __pycache__/scrapscript.*.pyc Lib/scrapscript.pyc
17+
cp style.css repl.html Lib
1518
cp python scrapscript.com
1619
./zip -r scrapscript.com Lib .args
1720
echo "Testing..."

‎fly.toml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# fly.toml app configuration file generated for scrapscript on 2023-11-29T02:06:16-05:00
2+
#
3+
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
4+
#
5+
6+
app = "scrapscript"
7+
primary_region = "ewr"
8+
9+
[build]
10+
11+
[http_service]
12+
internal_port = 8000
13+
force_https = true
14+
auto_stop_machines = true
15+
auto_start_machines = true
16+
min_machines_running = 0
17+
processes = ["app"]
18+
19+
[[vm]]
20+
cpu_kind = "shared"
21+
cpus = 1
22+
memory_mb = 256
23+
24+
[experimental]
25+
cmd = ["serve", "--port", "8000", "--debug", "--fork"]

‎repl.html‎

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
<!DOCTYPE html>
2+
<html>
3+
<head>
4+
<meta charset="utf-8">
5+
<meta name="viewport" content="width=device-width, initial-scale=1">
6+
<title>Scrapscript Web REPL</title>
7+
<link rel="shortcut icon" type="image/x-icon" href="data:image/vnd.microsoft.icon;base64,AAABAAEAEBAQAAAAAAAoAQAAFgAAACgAAAAQAAAAIAAAAAEABAAAAAAAgAAAAAAAAAAAAAAAEAAAAAAAAAD/AAAA////AAAA/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAREBEQAAAAAQAQEAEAAAABABAQAQAAAAAREBEQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACIiIiIiIiIiIiIiIiIiIiIiERIiESEiIiIiESEiISIiIhEiISIhEiIiIREiESEhIiIiIiIiIiIiIiIiIiIiIiIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA">
8+
<link rel="preconnect" href="https://fonts.googleapis.com" />
9+
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
10+
<link
11+
href="https://fonts.googleapis.com/css2?family=Fira+Code&display=swap"
12+
rel="stylesheet"
13+
/>
14+
<link
15+
href="https://fonts.googleapis.com/css2?family=Rubik:ital,wght@0,500;0,700&display=swap"
16+
rel="stylesheet"
17+
/>
18+
<link
19+
href="https://fonts.googleapis.com/css2?family=Nunito+Sans:ital,wght@0,400;0,700;0,900;1,400&display=swap"
20+
rel="stylesheet"
21+
/>
22+
<link rel="stylesheet" href="/style.css" />
23+
</head>
24+
<body>
25+
<main>
26+
<div>
27+
<p>See <a href="https://scrapscript.org/">scrapscript.org</a> for a slightly
28+
out of date language reference.</p>
29+
<p>This REPL sends programs to a server to be evaluated, but the server is
30+
completely stateless. All persistence is in the browser.</p>
31+
</div>
32+
<div>
33+
<!-- TODO(max): Add button to save to/load from disk. -->
34+
<button id="clear-local-storage">Clear LocalStorage</button>
35+
</div>
36+
<div id="output" style="height: 400px; overflow: auto;">
37+
Output:
38+
</div>
39+
<div>
40+
<code>>>> </code><input id="input" type="text" />
41+
</div>
42+
<script type="module">
43+
"use strict";
44+
45+
function updateHistory(inp, out) {
46+
const pre_inp = document.createElement("pre");
47+
pre_inp.setAttribute("class", "language-text");
48+
const code_inp = document.createElement("code");
49+
code_inp.setAttribute("class", "language-text");
50+
code_inp.append(`>>> ${inp}`);
51+
pre_inp.append(code_inp);
52+
53+
const pre_out = document.createElement("pre");
54+
pre_out.setAttribute("class", "result language-text");
55+
const code_out = document.createElement("code");
56+
code_out.setAttribute("class", "language-text");
57+
code_out.append(`${out}`);
58+
pre_out.append(code_out);
59+
60+
output.append(pre_inp);
61+
output.append(pre_out);
62+
output.scrollTop = output.scrollHeight;
63+
}
64+
65+
async function sendRequest(env, exp) {
66+
const params = env === null ? {exp} : {exp, env};
67+
return fetch("/eval?" + new URLSearchParams(params));
68+
}
69+
70+
let input = document.getElementById("input");
71+
const output = document.getElementById("output");
72+
const history = [];
73+
const button = document.getElementById("clear-local-storage");
74+
75+
function renderHistory() {
76+
for (const [inp, out] of history) {
77+
updateHistory(inp, out);
78+
}
79+
}
80+
81+
function loadFromLocalStorage() {
82+
history.length = 0;
83+
output.innerHTML = "";
84+
const hist = window.localStorage.getItem('history');
85+
if (hist !== null) {
86+
history.push(...JSON.parse(hist));
87+
}
88+
renderHistory();
89+
document.env = window.localStorage.getItem('env');
90+
}
91+
92+
function expandInput() {
93+
if (input.tagName === "TEXTAREA") {
94+
return;
95+
}
96+
const textarea = document.createElement("textarea");
97+
textarea.setAttribute("id", "input");
98+
textarea.setAttribute("rows", "1");
99+
textarea.setAttribute("style", "height: 1.5em;");
100+
textarea.value = input.value;
101+
input.replaceWith(textarea);
102+
input = textarea;
103+
input.focus();
104+
input.addEventListener("keyup", e => inputHandler(e));
105+
}
106+
107+
function shrinkInput() {
108+
if (input.tagName === "INPUT") {
109+
return;
110+
}
111+
const newInput = document.createElement("input");
112+
newInput.setAttribute("id", "input");
113+
newInput.setAttribute("type", "text");
114+
newInput.value = input.value;
115+
input.replaceWith(newInput);
116+
input = newInput;
117+
input.focus();
118+
input.addEventListener("keyup", e => inputHandler(e));
119+
}
120+
121+
loadFromLocalStorage();
122+
input.focus();
123+
async function inputHandler(e) {
124+
// TODO(max): Make up/down arrow keys navigate history
125+
if (e.key === "Enter") {
126+
if (e.shiftKey && input.tagName === "INPUT") {
127+
// Shift-Enter expands the input to a textarea and does not submit
128+
// input.
129+
expandInput();
130+
return;
131+
}
132+
if (!e.shiftKey && input.tagName === "TEXTAREA") {
133+
// Normal Enter in a textarea should not submit input.
134+
return;
135+
}
136+
const response = await sendRequest(document.env, input.value);
137+
if (response.ok) {
138+
const {env, result} = await response.json();
139+
updateHistory(input.value, result);
140+
history.push([input.value, result]);
141+
input.value = "";
142+
document.env = env;
143+
window.localStorage.setItem('env', env)
144+
window.localStorage.setItem('history', JSON.stringify(history));
145+
} else {
146+
const msg = await response.text();
147+
updateHistory(input.value, msg);
148+
input.value = "";
149+
}
150+
if (input.tagName === "TEXTAREA") {
151+
shrinkInput();
152+
}
153+
}
154+
}
155+
input.addEventListener("keyup", e => inputHandler(e));
156+
button.addEventListener("click", () => {
157+
window.localStorage.clear();
158+
loadFromLocalStorage();
159+
input.focus();
160+
});
161+
</script>
162+
</main>
163+
<script data-goatcounter="https://scrapscript.goatcounter.com/count"
164+
async src="//gc.zgo.at/count.js"></script>
165+
</body>
166+
</html>

‎scrapscript.py‎

Lines changed: 130 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,18 +4,21 @@
44
import code
55
import dataclasses
66
import enum
7+
import http.server
78
import json
89
import logging
910
import os
1011
import re
12+
import socketserver
1113
import sys
1214
import typing
1315
import unittest
1416
import urllib.request
17+
import urllib.parse
1518
from dataclasses import dataclass
1619
from enum import auto
1720
from types import FunctionType, ModuleType
18-
from typing import Any, Callable, Dict, Mapping, Optional, Union
21+
from typing import Any, Callable, Dict, Mapping, Optional, Tuple, Union
1922

2023
readline: Optional[ModuleType]
2124
try:
@@ -1320,6 +1323,96 @@ def deserialize(msg: str) -> Object:
13201323
return Object.deserialize(decoded)
13211324

13221325

1326+
class ScrapMonad:
1327+
def __init__(self, env: Env) -> None:
1328+
assert isinstance(env, dict) # for .copy()
1329+
self.env: Env = env.copy()
1330+
1331+
def bind(self, exp: Object) -> Tuple[Object, "ScrapMonad"]:
1332+
env = self.env
1333+
result = eval_exp(env, exp)
1334+
if isinstance(result, EnvObject):
1335+
return result, ScrapMonad({**env, **result.env})
1336+
return result, ScrapMonad({**env, "_": result})
1337+
1338+
1339+
ASSET_DIR = os.path.dirname(__file__)
1340+
1341+
1342+
class ScrapReplServer(http.server.SimpleHTTPRequestHandler):
1343+
def do_GET(self) -> None:
1344+
logger.debug("GET %s", self.path)
1345+
parsed_path = urllib.parse.urlsplit(self.path)
1346+
query = urllib.parse.parse_qs(parsed_path.query)
1347+
logging.debug("PATH %s", parsed_path)
1348+
logging.debug("QUERY %s", query)
1349+
if parsed_path.path == "/repl":
1350+
return self.do_repl()
1351+
if parsed_path.path == "/eval":
1352+
try:
1353+
return self.do_eval(query)
1354+
except Exception as e:
1355+
self.send_response(400)
1356+
self.send_header("Content-type", "text/plain")
1357+
self.end_headers()
1358+
self.wfile.write(str(e).encode("utf-8"))
1359+
return
1360+
if parsed_path.path == "/style.css":
1361+
self.send_response(200)
1362+
self.send_header("Content-type", "text/css")
1363+
self.end_headers()
1364+
with open(os.path.join(ASSET_DIR, "style.css"), "rb") as f:
1365+
self.wfile.write(f.read())
1366+
return
1367+
return self.do_404()
1368+
1369+
def do_repl(self) -> None:
1370+
self.send_response(200)
1371+
self.send_header("Content-type", "text/html")
1372+
self.end_headers()
1373+
with open(os.path.join(ASSET_DIR, "repl.html"), "rb") as f:
1374+
self.wfile.write(f.read())
1375+
return
1376+
1377+
def do_404(self) -> None:
1378+
self.send_response(404)
1379+
self.send_header("Content-type", "text/html")
1380+
self.end_headers()
1381+
self.wfile.write(b"""try hitting <a href="/repl">/repl</a>""")
1382+
return
1383+
1384+
def do_eval(self, query: Dict[str, Any]) -> None:
1385+
exp = query.get("exp")
1386+
if exp is None:
1387+
raise TypeError("Need expression to evaluate")
1388+
if len(exp) != 1:
1389+
raise TypeError("Need exactly one expression to evaluate")
1390+
exp = exp[0]
1391+
tokens = tokenize(exp)
1392+
ast = parse(tokens)
1393+
env = query.get("env")
1394+
if env is None:
1395+
env = STDLIB
1396+
else:
1397+
if len(env) != 1:
1398+
raise TypeError("Need exactly one env")
1399+
env_object = deserialize(env[0])
1400+
assert isinstance(env_object, EnvObject)
1401+
env = env_object.env
1402+
logging.debug("env is %s", env)
1403+
monad = ScrapMonad(env)
1404+
result, next_monad = monad.bind(ast)
1405+
serialized = EnvObject(next_monad.env).serialize()
1406+
encoded = bencode(serialized)
1407+
response = {"env": encoded.decode("utf-8"), "result": str(result)}
1408+
self.send_response(200)
1409+
self.send_header("Content-type", "text/plain")
1410+
self.send_header("Cache-Control", "max-age=3600")
1411+
self.end_headers()
1412+
self.wfile.write(json.dumps(response).encode("utf-8"))
1413+
return
1414+
1415+
13231416
class TokenizerTests(unittest.TestCase):
13241417
def test_tokenize_digit(self) -> None:
13251418
self.assertEqual(tokenize("1"), [IntLit(1)])
@@ -3918,6 +4011,21 @@ def test_serialize_function(self) -> None:
39184011
)
39194012

39204013

4014+
class ScrapMonadTests(unittest.TestCase):
4015+
def test_create_copies_env(self) -> None:
4016+
env = {"a": Int(123)}
4017+
result = ScrapMonad(env)
4018+
self.assertEqual(result.env, env)
4019+
self.assertIsNot(result.env, env)
4020+
4021+
def test_bind_returns_new_monad(self) -> None:
4022+
env = {"a": Int(123)}
4023+
orig = ScrapMonad(env)
4024+
result, next_monad = orig.bind(Assign(Var("b"), Int(456)))
4025+
self.assertEqual(orig.env, {"a": Int(123)})
4026+
self.assertEqual(next_monad.env, {"a": Int(123), "b": Int(456)})
4027+
4028+
39214029
class PrettyPrintTests(unittest.TestCase):
39224030
def test_pretty_print_int(self) -> None:
39234031
obj = Int(1)
@@ -4254,6 +4362,21 @@ def test_command(args: argparse.Namespace) -> None:
42544362
unittest.main(argv=[__file__, *args.unittest_args])
42554363

42564364

4365+
def serve_command(args: argparse.Namespace) -> None:
4366+
if args.debug:
4367+
logging.basicConfig(level=logging.DEBUG)
4368+
server: Union[type[socketserver.TCPServer], type[socketserver.ForkingTCPServer]]
4369+
if args.fork:
4370+
server = socketserver.ForkingTCPServer
4371+
else:
4372+
server = socketserver.TCPServer
4373+
server.allow_reuse_address = True
4374+
with server(("", args.port), ScrapReplServer) as httpd:
4375+
host, port = httpd.server_address
4376+
print(f"serving at http://{host!s}:{port}")
4377+
httpd.serve_forever()
4378+
4379+
42574380
def main() -> None:
42584381
parser = argparse.ArgumentParser(prog="scrapscript")
42594382
subparsers = parser.add_subparsers(dest="command")
@@ -4277,6 +4400,12 @@ def main() -> None:
42774400
apply.add_argument("program")
42784401
apply.add_argument("--debug", action="store_true")
42794402

4403+
serve = subparsers.add_parser("serve")
4404+
serve.set_defaults(func=serve_command)
4405+
serve.add_argument("--port", type=int, default=8000)
4406+
serve.add_argument("--debug", action="store_true")
4407+
serve.add_argument("--fork", action="store_true")
4408+
42804409
args = parser.parse_args()
42814410
if not args.command:
42824411
args.debug = False

0 commit comments

Comments
 (0)