diff --git a/ReadMe.md b/ReadMe.md index 1a43b4cf..323aef23 100644 --- a/ReadMe.md +++ b/ReadMe.md @@ -153,6 +153,9 @@ Execute as usual on Linux (`./hello`) or Windows (`hello`). For more details on compilation, see [**Compiling**](/doc/Compiling.md). +An optional Unix prototype, [**vloksh and shared modules**](/doc/SharedModules.md), +loads VOC modules on demand and provides a command shell with `.sym`-based completion. + ### Viewing the interfaces of included modules. In order to see the definition of a module's interface, use the "showdef" program. diff --git a/doc/SharedModules.md b/doc/SharedModules.md new file mode 100644 index 00000000..e5a1abf5 --- /dev/null +++ b/doc/SharedModules.md @@ -0,0 +1,158 @@ +# Shared modules and vloksh (Unix prototype) + +`vloksh` is a minimal VOC counterpart of polpo's loksh. It executes Oberon +`Module.Command` procedures in its own address space, loading native ELF `.so` +libraries on demand. The shell and loader are optional: the normal compiler +build and the installed runtime are unchanged. + +## Try it + +Requires an installed VOC with its shared runtime and development headers, +a C compiler, GNU Readline development files, Python 3 and binutils. +The current shell uses the default Oberon-2 type model (`-O2`). + +```sh +make vloksh +make vloksh-test +make vloksh-demo +build/vloksh/vloksh -Pbuild/vloksh/strutils-demo +``` + +The demo reads sources from `../strutils` without modifying that repository. +Override its location with `STRUTILS=/path/to/strutils`. +Inside the shell: + +```text +> StrDemo + StrDemo.Echo + StrDemo.Run +> StrDemo.Run +shared +modules +really +work +calls: 1 +> StrDemo.Run +shared +modules +really +work +calls: 2 +> StrDemo.Echo hello world +hello world +> quit +``` + +Tab completes module names, then their commands. Readline supplies editing, +session history and filename completion for arguments. Module and command +discovery does **not** load a library or run a module initializer. It reads the +existing `.sym` files, selecting exported, parameterless proper procedures, +as showdef's `OPT.Import` reader would. Already initialized modules are also +discoverable through VOC's command registry. + +The shell also supports single-command and batch operation: + +```sh +build/vloksh/vloksh -Pbuild/vloksh/strutils-demo StrDemo.Run +printf 'StrDemo.Run\nStrDemo.Run\n' | build/vloksh/vloksh -Pbuild/vloksh/strutils-demo +``` + +`-Ppath` or `-P path` overrides `VOC_MODULE_PATH`. Paths are colon-separated; +by default the loader searches the current directory and the directory of the +actually loaded `libvoc-O2.so`. Symbols are searched in those directories, +then `VOC_SYM_PATH`, or the installation's `2/sym` directory when it is unset. +`VOCROOT` can override the installation root for symbol discovery. + +Build overrides for another installation: + +```sh +make vloksh VOC=/opt/voc/bin/voc VOCROOT=/opt/voc VOCLIBDIR=/opt/voc/lib +``` + +Build products stay in `build/vloksh`. Nothing is installed by these targets. + +## Building and packaging modules + +The experimental helper builds **one module per shared object**, in import +order, in the current build directory: + +```sh +python3 /path/to/compiler/src/tools/vloksh/voc-shared.py /path/to/strTypes.Mod +python3 /path/to/compiler/src/tools/vloksh/voc-shared.py /path/to/strUtils.Mod +python3 /path/to/compiler/src/tools/vloksh/voc-shared.py /path/to/StrDemo.Mod +``` + +For module `M` it produces `libvoc-M-O2.so`, plus the usual `M.sym`, `M.h` and +generated C. It translates with VOC and compiles explicitly with `-fPIC`. +Every external import becomes an ELF `DT_NEEDED` dependency; modules already +provided by `libvoc-O2.so` are not duplicated. `$ORIGIN` run paths let libraries +in the same directory find each other. Imports in another directory need a +system linker search path or `LD_LIBRARY_PATH` at runtime; `-P` locates the +requested module but cannot override the ELF linker's dependency search. + +The helper supports `--root`, `--libdir`, `--voc`, `--cc`, `--model C`, `-I` +(additional C includes) and `-L` (shared dependencies). `OBERON`/`MODULES` +remain VOC's symbol search configuration. It honors `CC`, `CFLAGS`, +`LDFLAGS`, `LDLIBS`, `VOCROOT` and `VOCLIBDIR`. + +A Gentoo package can install: + +| File | Location / purpose | +| --- | --- | +| `libvoc-strTypes-O2.so`, `libvoc-strUtils-O2.so` | `/usr/$(get_libdir)`; runtime libraries | +| `strTypes.sym`, `strUtils.sym` | `/usr/share/voc/2/sym`; compilation and shell completion | +| `strTypes.h`, `strUtils.h` | `/usr/share/voc/2/include`; C backend compilation | + +Static archives can coexist for existing consumers; `.sym` and `.h` are still +needed when compiling against a packaged module. Runtime execution itself +does not need symbols, but completion of an unloaded module does. No command +index or other completion metadata is required. The overlay ebuilds have not +been changed by this experiment. + +## Loader contract and limitations + +- `SharedModules.ThisMod` opens `libvoc-M-O2.so` with + `dlopen(RTLD_NOW | RTLD_GLOBAL)`, checks a small ABI descriptor and calls + `M__init`. That initializer registers the module, its commands and GC roots + in the existing shared VOC runtime. `ThisCommand` uses the existing registry. + Existing `Modules.ThisMod` is unchanged; clients needing on-demand lookup + use `SharedModules.ThisMod` in this prototype. +- Handles and pointer-sized values use `SYSTEM.ADDRESS`, not `LONGINT`. +- The helper embeds `M__voc_abi`, checking pointer/basic-type sizes and runtime + descriptor sizes. This is a **shape check, not a complete ABI/versioning or + imported-interface fingerprint check**. Use matching compiler/runtime, + architecture, type model and dependency interfaces. Unrelated Ofront `.so` + files are not compatible. Dependencies must be built with the same contract. +- The shell and all loaded modules must use **one shared** `libvoc-O2.so`. + Statically embedding separate runtime/heap copies in plugins is unsafe. +- Libraries remain loaded until process exit. The loader pins a directly loaded + module's registry entry so `Modules.Free` cannot remove its GC roots. Imported + modules remain referenced by their clients. There is no unload/reload yet: + VOC's static type descriptors, pointer enumerators and finalizers can outlive + a command, making a casual `dlclose` unsafe. +- Commands take no formal parameters or result; arguments are available as + `Oberon.Par.text` starting at `Oberon.Par.pos`. General exported functions + remain callable by importing modules, but are not shell commands. +- HALT, assertions and faults terminate the prototype; there is no loksh-style + trap recovery, process isolation, Unix pipeline syntax or job control yet. + Load only trusted native modules: they have the shell's full permissions. +- The existing registry limits module names to 19 and command names to 23 + characters. The helper rejects names that would be truncated. + +## Layout and later V4 work + +`src/library/v4/SharedModules.Mod` is the reusable, UI-independent loader; +`CommandSymbols.Mod` is a read-only `.sym` command reader following +`src/compiler/OPM.Mod` and `OPT.Mod` (format `F7 84`). Unsupported or malformed +symbols return no partial completions. The small C support layer handles +POSIX library/path operations. The shell and Readline bridge live separately +in `src/tools/vloksh`. + +A later graphical port fits under `src/library/v4/ui`, rather than mixing +UI dependencies into the existing headless library. Its entry command could +be launched by the same loader. However, VOC already supplies headless +`Texts` and `Oberon` modules: a loaded UI cannot simply replace those modules +with incompatible records/interfaces of the same names. The port must first +unify the common interfaces or use distinct CLI/UI modules and import aliases, +as polpo does. X11 bindings, fonts/text resources, event loop and trap handling +also need adaptation. This prototype establishes loading, not that UI port. diff --git a/makefile b/makefile index f895559f..eb06f3b5 100644 --- a/makefile +++ b/makefile @@ -113,6 +113,17 @@ usage: tags: ctags -R --options=oberon.ctags --extras=+q +# Optional Unix command shell; does not rebuild or install the compiler. +.PHONY: vloksh vloksh-test vloksh-demo +vloksh: + $(MAKE) -f src/tools/vloksh/Makefile all + +vloksh-test: + $(MAKE) -f src/tools/vloksh/Makefile test + +vloksh-demo: + $(MAKE) -f src/tools/vloksh/Makefile demo + # Generate config files Configuration.Make and Configuration.Mod FORCE: diff --git a/src/library/v4/CommandSymbols.Mod b/src/library/v4/CommandSymbols.Mod new file mode 100644 index 00000000..e2bbc121 --- /dev/null +++ b/src/library/v4/CommandSymbols.Mod @@ -0,0 +1,175 @@ +MODULE CommandSymbols; + + (* Read-only command discovery from VOC .sym files (OPM/OPT format F7 84). + Like showdef, select exported XProc objects with no result/parameters. + Unlike OPT.Import, this reader builds no compiler symbol table and never + rewrites imported symbols. Keep the tags in sync with compiler/OPT.Mod. *) + + IMPORT SYSTEM, Files; + + TYPE + Visitor* = PROCEDURE(name: ARRAY OF CHAR); + Reader = RECORD + rider: Files.Rider; + length: LONGINT; + valid: BOOLEAN; + depth, refs: INTEGER + END; + Name = ARRAY 256 OF CHAR; + Entry = POINTER TO EntryDesc; + EntryDesc = RECORD next: Entry; name: Name END; + + PROCEDURE Byte(VAR r: Reader): CHAR; + VAR ch: CHAR; + BEGIN + IF ~r.valid OR (Files.Pos(r.rider) >= r.length) THEN + r.valid := FALSE; RETURN 0X + END; + Files.Read(r.rider, ch); + IF r.rider.eof THEN r.valid := FALSE END; + RETURN ch + END Byte; + + PROCEDURE Number(VAR r: Reader): SYSTEM.INT64; + VAR n, wide: SYSTEM.INT64; shift, b: INTEGER; + BEGIN + n := 0; shift := 0; + LOOP + b := ORD(Byte(r)); + IF ~r.valid THEN RETURN 0 END; + IF b < 128 THEN + IF (shift = 63) & (b # 0) & (b # 127) THEN r.valid := FALSE; RETURN 0 END; + wide := b MOD 64 - b DIV 64 * 64; + RETURN n + ASH(wide, shift) + END; + IF shift >= 63 THEN r.valid := FALSE; RETURN 0 END; + wide := b - 128; n := n + ASH(wide, shift); + INC(shift, 7) + END + END Number; + + PROCEDURE ReadName(VAR r: Reader; VAR name: Name); + VAR i: INTEGER; ch: CHAR; + BEGIN + i := 0; + REPEAT + ch := Byte(r); + IF i >= LEN(name) - 1 THEN r.valid := FALSE; name[0] := 0X; RETURN END; + name[i] := ch; INC(i) + UNTIL ~r.valid OR (ch = 0X) + END ReadName; + + PROCEDURE Skip(VAR r: Reader; count: SYSTEM.INT64); + BEGIN + IF (count < 0) OR (count > r.length - Files.Pos(r.rider)) THEN r.valid := FALSE + ELSE Files.Set(r.rider, Files.Base(r.rider), Files.Pos(r.rider) + SHORT(count)) + END + END Skip; + + PROCEDURE ^Structure(VAR r: Reader): BOOLEAN; + + PROCEDURE Signature(VAR r: Reader): BOOLEAN; + VAR noResult, noParams, ignored: BOOLEAN; tag, n: SYSTEM.INT64; name: Name; + BEGIN + noResult := Structure(r); tag := Number(r); noParams := tag = 18; + WHILE r.valid & (tag # 18) DO + IF (tag # 23) & (tag # 24) & (tag # 42) THEN r.valid := FALSE; RETURN FALSE END; + ignored := Structure(r); n := Number(r); ReadName(r, name); tag := Number(r) + END; + RETURN r.valid & noResult & noParams + END Signature; + + PROCEDURE Structure(VAR r: Reader): BOOLEAN; + VAR tag, n: SYSTEM.INT64; ignored: BOOLEAN; name: Name; + BEGIN + tag := Number(r); + IF tag # 34 THEN + IF (tag >= 0) OR (-tag >= r.refs) THEN r.valid := FALSE END; + IF (tag = -4) OR (tag = -7) THEN n := Number(r) END; + RETURN r.valid & (tag = -10) (* OPT.NoTyp *) + END; + INC(r.depth); INC(r.refs); + IF (r.depth > 128) OR (r.refs > 16384) THEN r.valid := FALSE; RETURN FALSE END; + tag := Number(r); (* module index or Smname *) + IF tag = 16 THEN ReadName(r, name) + ELSIF tag > 0 THEN r.valid := FALSE + END; + ReadName(r, name); tag := Number(r); + IF tag = 35 THEN n := Number(r); tag := Number(r) END; (* Ssys *) + CASE SHORT(tag) OF + | 36, 38: ignored := Structure(r) (* pointer, dynamic array *) + | 37: ignored := Structure(r); n := Number(r) (* fixed array *) + | 40: ignored := Signature(r) (* procedure type *) + | 39: (* record: base, size, alignment, method count, fields/methods *) + ignored := Structure(r); + n := Number(r); n := Number(r); n := Number(r); tag := Number(r); + WHILE r.valid & (tag # 18) DO + CASE SHORT(tag) OF + | 25, 26: ignored := Structure(r); ReadName(r, name); n := Number(r) + | 27, 28, 30: n := Number(r) + | 29: ignored := Signature(r); ReadName(r, name); n := Number(r) + ELSE r.valid := FALSE + END; + tag := Number(r) + END + ELSE r.valid := FALSE + END; + DEC(r.depth); + RETURN FALSE + END Structure; + + PROCEDURE Enumerate*(filename, module: ARRAY OF CHAR; visit: Visitor): BOOLEAN; + VAR f: Files.File; r: Reader; name: Name; tag, n: SYSTEM.INT64; + command, ignored: BOOLEAN; first, last, entry: Entry; ch: CHAR; + BEGIN + f := Files.Old(filename); + IF f = NIL THEN RETURN FALSE END; + Files.Set(r.rider, f, 0); r.length := Files.Length(f); + r.valid := TRUE; r.depth := 0; r.refs := 14; first := NIL; last := NIL; + ch := Byte(r); IF ch # 0F7X THEN r.valid := FALSE END; + ch := Byte(r); IF ch # 084X THEN r.valid := FALSE END; + tag := Number(r); IF tag # 16 THEN r.valid := FALSE END; + ReadName(r, name); IF name # module THEN r.valid := FALSE END; + (* Link names precede the exported objects. *) + REPEAT ReadName(r, name) UNTIL ~r.valid OR (name = ""); + WHILE r.valid & (Files.Pos(r.rider) < r.length) DO + tag := Number(r); command := FALSE; + WHILE r.valid & (tag = 41) DO (* exported documentation *) + n := Number(r); Skip(r, n); tag := Number(r) + END; + IF (tag >= 1) & (tag <= 9) THEN + CASE SHORT(tag) OF + | 1, 2, 3: ch := Byte(r) + | 4, 7: n := Number(r); n := Number(r) + | 5: Skip(r, 4) + | 6: Skip(r, 8) + | 8: ReadName(r, name) + | 9: (* NIL *) + ELSE r.valid := FALSE + END; + ReadName(r, name) + ELSIF tag = 19 THEN ignored := Structure(r) + ELSIF (tag = 20) OR (tag = 21) OR (tag = 22) THEN + ignored := Structure(r); ReadName(r, name) + ELSIF (tag >= 31) & (tag <= 33) THEN + command := Signature(r) & (tag = 31); + IF tag = 33 THEN n := Number(r); Skip(r, n) END; + ReadName(r, name); + IF command & r.valid THEN + NEW(entry); entry.name := name; + IF last = NIL THEN first := entry ELSE last.next := entry END; + last := entry + END + ELSE r.valid := FALSE + END + END; + Files.Close(f); + (* No partial results from an unsupported or corrupt symbol file. *) + IF r.valid THEN + entry := first; + WHILE entry # NIL DO visit(entry.name); entry := entry.next END + END; + RETURN r.valid + END Enumerate; + +END CommandSymbols. diff --git a/src/library/v4/SharedModules.Mod b/src/library/v4/SharedModules.Mod new file mode 100644 index 00000000..5df6aaa8 --- /dev/null +++ b/src/library/v4/SharedModules.Mod @@ -0,0 +1,107 @@ +MODULE SharedModules; + + (* Optional Unix shared-object loader. Uses VOC's existing module registry. + Libraries remain mapped for the lifetime of the process: the heap can + retain their enumPtrs procedures, type descriptors and finalizers. *) + + IMPORT SYSTEM, Heap, Modules, CommandSymbols; + + TYPE + Module* = Modules.Module; + Command* = Modules.Command; + Visitor* = CommandSymbols.Visitor; + ModuleBody = PROCEDURE(): SYSTEM.PTR; + + VAR + path*: ARRAY 4096 OF CHAR; + res*: INTEGER; + resMsg*: ARRAY 1024 OF CHAR; + model: ARRAY 3 OF CHAR; + + PROCEDURE -AInclude '#include "SharedModulesSupport.h"'; + + PROCEDURE -DefaultPath(VAR search: ARRAY OF CHAR) + 'VOCShared_default_path((char*)search, search__len)'; + + PROCEDURE -Open(name, search, model: ARRAY OF CHAR; + shortSize, intSize, longSize, setSize: SYSTEM.INT32; + VAR error: ARRAY OF CHAR): SYSTEM.ADDRESS + 'VOCShared_open((char*)name, (char*)search, (char*)model, shortSize, intSize, longSize, setSize, (char*)error, error__len)'; + + PROCEDURE -Body(handle: SYSTEM.ADDRESS; name: ARRAY OF CHAR): ModuleBody + '(SharedModules_ModuleBody)VOCShared_body(handle, (char*)name)'; + + PROCEDURE -DiskModules(search, model: ARRAY OF CHAR; visit: Visitor) + 'VOCShared_modules((char*)search, (char*)model, visit)'; + + PROCEDURE -SymbolFile(name, search, model: ARRAY OF CHAR; + VAR filename: ARRAY OF CHAR): BOOLEAN + 'VOCShared_symbol((char*)name, (char*)search, (char*)model, (char*)filename, filename__len)'; + + PROCEDURE First*(): Module; + BEGIN RETURN SYSTEM.VAL(Module, Heap.modules) + END First; + + PROCEDURE ThisMod*(name: ARRAY OF CHAR): Module; + VAR m: Module; handle: SYSTEM.ADDRESS; body: ModuleBody; + BEGIN + m := Modules.ThisMod(name); + IF m = NIL THEN + handle := Open(name, path, model, SIZE(SHORTINT), SIZE(INTEGER), + SIZE(LONGINT), SIZE(SET), resMsg); + IF handle = 0 THEN res := 1; RETURN NIL END; + body := Body(handle, name); + m := SYSTEM.VAL(Module, body()); + IF (m = NIL) OR (m.name # name) THEN + (* Do not unmap code after calling an initializer, even on failure. *) + res := 1; resMsg := "initializer did not register the requested module"; + RETURN NIL + END; + Heap.INCREF(m) (* Pin its registry/GC roots; Modules.Free must not remove them. *) + END; + res := 0; resMsg := ""; + RETURN m + END ThisMod; + + PROCEDURE ThisCommand*(m: Module; name: ARRAY OF CHAR): Command; + VAR cmd: Command; + BEGIN + IF m = NIL THEN res := 1; resMsg := "module not loaded"; RETURN NIL END; + cmd := Modules.ThisCommand(m, name); + res := Modules.res; COPY(Modules.resMsg, resMsg); + RETURN cmd + END ThisCommand; + + PROCEDURE EnumerateModules*(visit: Visitor); + VAR m: Module; + BEGIN + m := First(); + WHILE m # NIL DO visit(m.name); m := m.next END; + DiskModules(path, model, visit) + END EnumerateModules; + + PROCEDURE EnumerateCommands*(name: ARRAY OF CHAR; visit: Visitor): BOOLEAN; + VAR m: Module; c: Modules.Cmd; filename: ARRAY 8192 OF CHAR; valid: BOOLEAN; + BEGIN + m := Modules.ThisMod(name); + IF m # NIL THEN + c := m.cmds; + WHILE c # NIL DO visit(c.name); c := c.next END; + res := 0; resMsg := ""; RETURN TRUE + ELSE + (* Read symbols, without dlopen or module initialization. *) + IF SymbolFile(name, path, model, filename) THEN + valid := CommandSymbols.Enumerate(filename, name, visit); + IF valid THEN res := 0; resMsg := ""; RETURN TRUE END; + resMsg := "invalid or unsupported symbol file" + ELSE + resMsg := "symbol file not found (set VOC_SYM_PATH if symbols are elsewhere)" + END + END; + res := 1; RETURN FALSE + END EnumerateCommands; + +BEGIN + IF SIZE(INTEGER) = 2 THEN model := "O2" ELSE model := "OC" END; + DefaultPath(path) +END SharedModules. diff --git a/src/library/v4/SharedModulesSupport.c b/src/library/v4/SharedModulesSupport.c new file mode 100644 index 00000000..153b4b4a --- /dev/null +++ b/src/library/v4/SharedModulesSupport.c @@ -0,0 +1,211 @@ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include "SharedModulesSupport.h" + +/* The handles returned by open are deliberately never closed after __init. + In particular, dlclose is NOT an implementation of Heap.FreeModule. */ + +static int identifier(const char *s, size_t maximum) +{ + size_t n = 0; + if (!((*s >= 'A' && *s <= 'Z') || (*s >= 'a' && *s <= 'z'))) return 0; + for (; *s; s++, n++) { + if (!((*s >= 'A' && *s <= 'Z') || (*s >= 'a' && *s <= 'z') || + (*s >= '0' && *s <= '9') || *s == '_')) return 0; + } + return n <= maximum; +} + +/* Empty path components mean the current directory, as in loksh. */ +static int directory(const char **path, char *dir, size_t capacity) +{ + const char *end; + size_t n; + if (!*path) return 0; + end = strchr(*path, ':'); + n = end ? (size_t)(end - *path) : strlen(*path); + if (n >= capacity) { *path = end ? end + 1 : NULL; return -1; } + memcpy(dir, *path, n); + dir[n] = 0; + if (!n) strcpy(dir, "."); + *path = end ? end + 1 : NULL; + return 1; +} + +static int filename(char *out, size_t capacity, const char *dir, + const char *name, const char *model, const char *extension) +{ + int n = snprintf(out, capacity, "%s/libvoc-%s-%s.%s", dir, name, model, extension); + return n >= 0 && (size_t)n < capacity; +} + +void VOCShared_default_path(char *path, ADDRESS capacity) +{ + Dl_info info; + char dir[4096], *slash; + const char *env = getenv("VOC_MODULE_PATH"); + if (env) { snprintf(path, (size_t)capacity, "%s", env); return; } + snprintf(path, (size_t)capacity, "."); + /* Locate the actual shared runtime, not the executable or a compiled-in + /usr/lib versus /usr/lib64 guess. */ + if (dladdr((void *)Heap_REGMOD, &info) && info.dli_fname) { + snprintf(dir, sizeof dir, "%s", info.dli_fname); + slash = strrchr(dir, '/'); + if (slash) { + *slash = 0; + snprintf(path, (size_t)capacity, ".:%s", dir); + } + } +} + +ADDRESS VOCShared_open(const char *name, const char *path, const char *model, + INT32 short_size, INT32 int_size, INT32 long_size, INT32 set_size, + char *error, ADDRESS capacity) +{ + char dir[4096], file[8192], symbol[128]; + const char *cursor = path; + void *handle = NULL; + const INT32 *abi; + const INT32 expected[] = {1, sizeof(ADDRESS), short_size, int_size, + long_size, set_size, sizeof(Heap_ModuleDesc), sizeof(Heap_CmdDesc)}; + size_t i; + int status, found = 0; + + if (!identifier(name, sizeof(Heap_ModuleName) - 1)) { + snprintf(error, (size_t)capacity, "invalid module name (maximum 19 characters)"); + return 0; + } + while ((status = directory(&cursor, dir, sizeof dir))) { + if (status < 0 || !filename(file, sizeof file, dir, name, model, "so")) continue; + if (access(file, F_OK) == 0) { + found = 1; + handle = dlopen(file, RTLD_NOW | RTLD_GLOBAL); + break; + } + } + if (!found) { + snprintf(file, sizeof file, "libvoc-%s-%s.so", name, model); + handle = dlopen(file, RTLD_NOW | RTLD_GLOBAL); + } + if (!handle) { + const char *message = dlerror(); + snprintf(error, (size_t)capacity, "%s", message ? message : "library not found"); + return 0; + } + + snprintf(symbol, sizeof symbol, "%s__voc_abi", name); + abi = dlsym(handle, symbol); + if (!abi) { + snprintf(error, (size_t)capacity, "%s: missing VOC shared-module ABI descriptor", file); + dlclose(handle); + return 0; + } + for (i = 0; i < sizeof expected / sizeof expected[0]; i++) { + if (abi[i] != expected[i]) { + snprintf(error, (size_t)capacity, "%s: incompatible VOC shared-module ABI", file); + dlclose(handle); + return 0; + } + } + snprintf(symbol, sizeof symbol, "%s__init", name); + if (!dlsym(handle, symbol)) { + snprintf(error, (size_t)capacity, "%s: missing %s", file, symbol); + dlclose(handle); + return 0; + } + error[0] = 0; + return (ADDRESS)handle; +} + +void *VOCShared_body(ADDRESS handle, const char *name) +{ + char symbol[128]; + snprintf(symbol, sizeof symbol, "%s__init", name); + return dlsym((void *)handle, symbol); +} + +static void symbol_path(char *path, size_t capacity, const char *model) +{ + Dl_info info; + char prefix[4096], *slash; + const char *env = getenv("VOC_SYM_PATH"); + const char *root = getenv("VOCROOT"); + const char *subdir = !strcmp(model, "O2") ? "2" : "C"; + if (env) { snprintf(path, capacity, "%s", env); return; } + if (root) { snprintf(path, capacity, "%s/%s/sym", root, subdir); return; } + path[0] = 0; + if (dladdr((void *)Heap_REGMOD, &info) && info.dli_fname) { + snprintf(prefix, sizeof prefix, "%s", info.dli_fname); + slash = strrchr(prefix, '/'); + if (!slash) return; + *slash = 0; + slash = strrchr(prefix, '/'); + if (!slash) return; + *slash = 0; + snprintf(path, capacity, "%s/%s/sym:%s/share/voc/%s/sym", prefix, subdir, prefix, subdir); + } +} + +static void enumerate_directory(const char *dir, const char *model, + VOCShared_Visitor visit, int symbols) +{ + char suffix[32], name[sizeof(Heap_ModuleName)]; + DIR *stream; + struct dirent *entry; + size_t n, prefix_size = symbols ? 0 : 7, suffix_size; + if (!(stream = opendir(dir))) return; + if (symbols) snprintf(suffix, sizeof suffix, ".sym"); + else snprintf(suffix, sizeof suffix, "-%s.so", model); + suffix_size = strlen(suffix); + while ((entry = readdir(stream))) { + n = strlen(entry->d_name); + if (n <= prefix_size + suffix_size || + (!symbols && strncmp(entry->d_name, "libvoc-", prefix_size)) || + strcmp(entry->d_name + n - suffix_size, suffix)) continue; + n -= prefix_size + suffix_size; + if (n >= sizeof name) continue; + memcpy(name, entry->d_name + prefix_size, n); + name[n] = 0; + if (identifier(name, sizeof name - 1)) visit((CHAR *)name, sizeof name); + } + closedir(stream); +} + +void VOCShared_modules(const char *path, const char *model, VOCShared_Visitor visit) +{ + char dir[4096], symbols[8192]; + const char *cursor = path; + int status; + while ((status = directory(&cursor, dir, sizeof dir))) { + if (status < 0) continue; + enumerate_directory(dir, model, visit, 0); + enumerate_directory(dir, model, visit, 1); + } + symbol_path(symbols, sizeof symbols, model); + cursor = symbols; + while ((status = directory(&cursor, dir, sizeof dir))) + if (status > 0) enumerate_directory(dir, model, visit, 1); +} + +int VOCShared_symbol(const char *name, const char *path, const char *model, + char *file, ADDRESS capacity) +{ + char dir[4096], symbols[8192]; + const char *cursor = path; + int pass, status, n; + if (!identifier(name, sizeof(Heap_ModuleName) - 1)) return 0; + symbol_path(symbols, sizeof symbols, model); + for (pass = 0; pass < 2; pass++, cursor = symbols) { + while ((status = directory(&cursor, dir, sizeof dir))) { + if (status < 0) continue; + n = snprintf(file, (size_t)capacity, "%s/%s.sym", dir, name); + if (n >= 0 && n < capacity && !access(file, R_OK)) return 1; + } + } + return 0; +} diff --git a/src/library/v4/SharedModulesSupport.h b/src/library/v4/SharedModulesSupport.h new file mode 100644 index 00000000..8c044b91 --- /dev/null +++ b/src/library/v4/SharedModulesSupport.h @@ -0,0 +1,17 @@ +#ifndef VOC_SHARED_MODULES_SUPPORT_H +#define VOC_SHARED_MODULES_SUPPORT_H + +#include "Heap.h" + +typedef void (*VOCShared_Visitor)(CHAR *name, ADDRESS length); + +void VOCShared_default_path(char *path, ADDRESS capacity); +ADDRESS VOCShared_open(const char *name, const char *path, const char *model, + INT32 short_size, INT32 int_size, INT32 long_size, INT32 set_size, + char *error, ADDRESS capacity); +void *VOCShared_body(ADDRESS handle, const char *name); +void VOCShared_modules(const char *path, const char *model, VOCShared_Visitor visit); +int VOCShared_symbol(const char *name, const char *path, const char *model, + char *filename, ADDRESS capacity); + +#endif diff --git a/src/tools/vloksh/Makefile b/src/tools/vloksh/Makefile new file mode 100644 index 00000000..52b5920a --- /dev/null +++ b/src/tools/vloksh/Makefile @@ -0,0 +1,34 @@ +# Optional Unix prototype, built against an existing VOC installation. +VOC ?= voc +CC ?= cc +VOCROOT ?= /usr/share/voc +VOCLIBDIR ?= $(dir $(shell $(CC) -print-file-name=libvoc-O2.so)) +ROOT := $(abspath $(dir $(lastword $(MAKEFILE_LIST)))/../../..) +BUILD ?= $(ROOT)/build/vloksh +SUPPORT := $(ROOT)/src/library/v4 +TOOLS := $(ROOT)/src/tools/vloksh +INCLUDES := -I$(VOCROOT)/2/include -I$(SUPPORT) -I$(TOOLS) -I$(BUILD) + +.PHONY: all test demo +all: $(BUILD)/vloksh + +$(BUILD): + mkdir -p "$@" + +$(BUILD)/CommandSymbols.c: $(SUPPORT)/CommandSymbols.Mod | $(BUILD) + cd "$(BUILD)" && VOCROOT="$(VOCROOT)" VOCLIBDIR="$(VOCLIBDIR)" $(VOC) -sFS "$(SUPPORT)/CommandSymbols.Mod" + +$(BUILD)/SharedModules.c: $(SUPPORT)/SharedModules.Mod $(BUILD)/CommandSymbols.c + cd "$(BUILD)" && VOCROOT="$(VOCROOT)" VOCLIBDIR="$(VOCLIBDIR)" $(VOC) -sFS "$(SUPPORT)/SharedModules.Mod" + +$(BUILD)/vloksh.c: $(TOOLS)/vloksh.Mod $(BUILD)/SharedModules.c + cd "$(BUILD)" && VOCROOT="$(VOCROOT)" VOCLIBDIR="$(VOCLIBDIR)" $(VOC) -sSm "$(TOOLS)/vloksh.Mod" + +$(BUILD)/vloksh: $(BUILD)/vloksh.c $(BUILD)/SharedModules.c $(BUILD)/CommandSymbols.c $(SUPPORT)/SharedModulesSupport.c $(SUPPORT)/SharedModulesSupport.h $(TOOLS)/Terminal.c $(TOOLS)/Terminal.h + $(CC) $(CFLAGS) -fPIC -g -Wno-stringop-overflow $(INCLUDES) $(BUILD)/vloksh.c $(BUILD)/SharedModules.c $(BUILD)/CommandSymbols.c $(SUPPORT)/SharedModulesSupport.c $(TOOLS)/Terminal.c $(LDFLAGS) -L$(VOCLIBDIR) -Wl,-rpath,$(abspath $(VOCLIBDIR)) -o "$@" -lvoc-O2 -lreadline -ldl + +test: all + VOC="$(VOC)" VOCROOT="$(VOCROOT)" VOCLIBDIR="$(VOCLIBDIR)" CC="$(CC)" python3 "$(TOOLS)/test.py" "$(BUILD)/vloksh" + +demo: all + VOC="$(VOC)" VOCROOT="$(VOCROOT)" VOCLIBDIR="$(VOCLIBDIR)" CC="$(CC)" python3 "$(TOOLS)/demo.py" "$(BUILD)" diff --git a/src/tools/vloksh/Terminal.c b/src/tools/vloksh/Terminal.c new file mode 100644 index 00000000..ad41005a --- /dev/null +++ b/src/tools/vloksh/Terminal.c @@ -0,0 +1,86 @@ +#include +#include +#include +#include +#include +#include +#include "Terminal.h" + +static VlokshComplete complete; +static char **candidates; +static size_t count, next; + +void Vloksh_add_completion(const char *name) +{ + char **grown; + char *copy; + size_t i; + for (i = 0; i < count; i++) if (!strcmp(candidates[i], name)) return; + copy = strdup(name); + if (!copy) return; + grown = realloc(candidates, (count + 1) * sizeof *candidates); + if (!grown) { free(copy); return; } + candidates = grown; + candidates[count++] = copy; +} + +static char *candidate(const char *text, int state) +{ + (void)text; + if (!state) next = 0; + return next < count ? strdup(candidates[next++]) : NULL; +} + +static char **matches(const char *text, int start, int end) +{ + char **result; + size_t i; + (void)end; + /* First word: Oberon commands. Arguments: Readline's file completion. */ + for (i = 0; i < (size_t)start; i++) + if (rl_line_buffer[i] != ' ' && rl_line_buffer[i] != '\t') return NULL; + rl_attempted_completion_over = 1; + /* A completed module ends in '.', not a space: the command follows. */ + rl_completion_suppress_append = strchr(text, '.') == NULL; + count = 0; + complete((CHAR *)text, (ADDRESS)strlen(text) + 1); + result = rl_completion_matches(text, candidate); + for (i = 0; i < count; i++) free(candidates[i]); + free(candidates); + candidates = NULL; + count = 0; + return result; +} + +void Vloksh_terminal_init(VlokshComplete callback) +{ + complete = callback; + rl_readline_name = "vloksh"; + rl_completer_word_break_characters = " \t\n"; + rl_attempted_completion_function = matches; + using_history(); + stifle_history(64); +} + +int Vloksh_read_line(char *line, ADDRESS capacity) +{ + char *input = NULL; + size_t allocated = 0, length; + ssize_t n; + if (isatty(STDIN_FILENO)) { + input = readline("> "); + if (!input) return 0; + if (*input) add_history(input); + length = strlen(input); + } else { + n = getline(&input, &allocated, stdin); + if (n < 0) { free(input); return 0; } + length = (size_t)n; + while (length && (input[length - 1] == '\n' || input[length - 1] == '\r')) --length; + input[length] = 0; + } + if (length >= (size_t)capacity) { free(input); return -1; } + memcpy(line, input, length + 1); + free(input); + return 1; +} diff --git a/src/tools/vloksh/Terminal.h b/src/tools/vloksh/Terminal.h new file mode 100644 index 00000000..c85bd2b0 --- /dev/null +++ b/src/tools/vloksh/Terminal.h @@ -0,0 +1,11 @@ +#ifndef VLOKSH_TERMINAL_H +#define VLOKSH_TERMINAL_H + +#include "SYSTEM.h" + +typedef void (*VlokshComplete)(CHAR *prefix, ADDRESS length); +void Vloksh_terminal_init(VlokshComplete complete); +int Vloksh_read_line(char *line, ADDRESS capacity); +void Vloksh_add_completion(const char *name); + +#endif diff --git a/src/tools/vloksh/demo.py b/src/tools/vloksh/demo.py new file mode 100644 index 00000000..801ade98 --- /dev/null +++ b/src/tools/vloksh/demo.py @@ -0,0 +1,19 @@ +#!/usr/bin/env python3 +"""Build the optional strutils experiment without modifying that repository.""" + +import os +from pathlib import Path +import subprocess +import sys + +tools = Path(__file__).resolve().parent +compiler = tools.parents[2] +strutils = Path(os.environ.get("STRUTILS", str(compiler.parent / "strutils"))) +output = Path(sys.argv[1]).resolve() / "strutils-demo" +output.mkdir(parents=True, exist_ok=True) +for source in [strutils / "src/strTypes.Mod", strutils / "src/strUtils.Mod", + tools / "examples/StrDemo.Mod"]: + subprocess.run([sys.executable, str(tools / "voc-shared.py"), str(source)], + cwd=output, check=True) +print(f"\nRun: {output.parent / 'vloksh'} -P{output} StrDemo.Run") +print(f"Or: {output.parent / 'vloksh'} -P{output} (then StrDemo.)") diff --git a/src/tools/vloksh/examples/StrDemo.Mod b/src/tools/vloksh/examples/StrDemo.Mod new file mode 100644 index 00000000..deb6f5a9 --- /dev/null +++ b/src/tools/vloksh/examples/StrDemo.Mod @@ -0,0 +1,28 @@ +MODULE StrDemo; + + (* A command module; neither it nor strUtils is linked into vloksh. *) + IMPORT strUtils, Out, Oberon, Texts; + + VAR count: LONGINT; + + PROCEDURE Run*; + VAR words: strUtils.pstrings; i: LONGINT; + BEGIN + words := strUtils.tokenize("shared modules really work", " "); + i := 0; + WHILE i < LEN(words^) DO + Out.String(words^[i]^); Out.Ln; INC(i) + END; + INC(count); + Out.String("calls: "); Out.Int(count, 0); Out.Ln + END Run; + + PROCEDURE Echo*; + VAR r: Texts.Reader; ch: CHAR; + BEGIN + Texts.OpenReader(r, Oberon.Par.text, Oberon.Par.pos); + WHILE ~r.eot DO Texts.Read(r, ch); IF ~r.eot THEN Out.Char(ch) END END; + Out.Ln + END Echo; + +END StrDemo. diff --git a/src/tools/vloksh/test.py b/src/tools/vloksh/test.py new file mode 100644 index 00000000..db0cdcc0 --- /dev/null +++ b/src/tools/vloksh/test.py @@ -0,0 +1,206 @@ +#!/usr/bin/env python3 +"""Integration tests using the installed VOC, ELF loader, and a real pty.""" + +import os +from pathlib import Path +import pty +import re +import select +import shlex +import subprocess +import sys +import tempfile +import time + +tools = Path(__file__).resolve().parent +shell = Path(sys.argv[1]).resolve() +helper = tools / "voc-shared.py" + +PROVIDER = '''MODULE Provider; +IMPORT Out; +VAR count: LONGINT; +PROCEDURE Next*(): LONGINT; +BEGIN INC(count); RETURN count END Next; +BEGIN Out.String("provider initialized"); Out.Ln +END Provider. +''' + +CONSUMER = '''MODULE Consumer; +IMPORT Provider, Out, Oberon, Texts, Heap; +CONST A* = 1234567890; B* = {1, 3}; C* = "string"; D* = 1.5; E* = NIL; +TYPE Node* = POINTER TO NodeDesc; + NodeDesc* = RECORD next*: Node; n*: LONGINT END; + Handler* = PROCEDURE (VAR node: Node): Node; + Grid* = ARRAY 3, 4 OF Node; +VAR kept*: Node; handler*: Handler; +PROCEDURE (node: Node) Touch*; +BEGIN INC(node.n) END Touch; +PROCEDURE Function*(): LONGINT; +BEGIN RETURN 17 END Function; +PROCEDURE WithArg*(n: LONGINT); +BEGIN Out.Int(n, 0) END WithArg; +(** A command documented in the symbol file. *) +PROCEDURE Run*; +BEGIN + IF kept = NIL THEN NEW(kept); kept.n := 40 END; + kept.Touch; Heap.GC(TRUE); + Out.String("run "); Out.Int(Provider.Next(), 0); + Out.String(" kept "); Out.Int(kept.n, 0); Out.Ln +END Run; +PROCEDURE Echo*; +VAR r: Texts.Reader; ch: CHAR; +BEGIN + Texts.OpenReader(r, Oberon.Par.text, Oberon.Par.pos); + WHILE ~r.eot DO Texts.Read(r, ch); IF ~r.eot THEN Out.Char(ch) END END; + Out.Ln +END Echo; +BEGIN Out.String("consumer initialized"); Out.Ln +END Consumer. +''' + + +def run(*args, input=None, expected=0, env=None): + result = subprocess.run([str(shell), *map(str, args)], input=input, text=True, + stdout=subprocess.PIPE, stderr=subprocess.STDOUT, + env=env, timeout=10) + assert result.returncode == expected, (args, result.returncode, result.stdout) + return result.stdout + + +def build(directory, name, source): + file = directory / f"{name}.Mod" + file.write_text(source) + subprocess.run([sys.executable, str(helper), str(file)], cwd=directory, check=True) + + +def interactive(directory): + pid, fd = pty.fork() + if pid == 0: + os.execv(str(shell), [str(shell), f"-P{directory}"]) + collected = bytearray() + + def until(needle): + data = bytearray() + deadline = time.monotonic() + 10 + while needle not in data: + remaining = deadline - time.monotonic() + assert remaining > 0, (needle, bytes(data)) + ready, _, _ = select.select([fd], [], [], remaining) + assert ready, (needle, bytes(data)) + chunk = os.read(fd, 65536) + assert chunk, bytes(data) + data.extend(chunk) + collected.extend(data) + return bytes(data) + + try: + until(b"> ") + # Completion reads symbols even when ELF dependencies cannot be loaded. + dependency = directory / "libvoc-Provider-O2.so" + hidden = dependency.with_suffix(".hidden") + dependency.rename(hidden) + try: + os.write(fd, b"Consu\t") + result = until(b"Consumer.") + assert b"initialized" not in result + os.write(fd, b"R\t") + result = until(b"Run") + assert b"initialized" not in result + finally: + hidden.rename(dependency) + os.write(fd, b"\n") + result = until(b"> ") + assert b"run 1 kept 41" in result, result + # History and in-process state survive the first call. + os.write(fd, b"\x1b[A\n") + result = until(b"> ") + assert b"run 2 kept 42" in result, result + os.write(fd, b"\x04") + _, status = os.waitpid(pid, 0) + assert os.waitstatus_to_exitcode(status) == 0 + pid = None + finally: + os.close(fd) + if pid: + os.kill(pid, 9) + os.waitpid(pid, 0) + + +def main(directory): + build(directory, "Provider", PROVIDER) + build(directory, "Consumer", CONSUMER) + path = f"-P{directory}" + listed = run(path, "Consumer") + assert set(listed.splitlines()) == {" Consumer.Echo", " Consumer.Run"}, listed + assert "initialized" not in listed + assert "Function" not in listed and "WithArg" not in listed and "Touch" not in listed + # Discovery does not require any shared object at all. + library = directory / "libvoc-Consumer-O2.so" + library.rename(library.with_suffix(".hidden")) + assert run(path, "Consumer") == listed + library.with_suffix(".hidden").rename(library) + + output = run(path, "Consumer.Run") + assert "provider initialized\nconsumer initialized\nrun 1 kept 41\n" == output, output + output = run(path, input="Consumer.Run\nConsumer.Run\nConsumer.Echo batch parameters\nquit\n") + assert output.count("consumer initialized") == 1, output + assert "run 1 kept 41\nrun 2 kept 42\nbatch parameters\n" in output, output + assert run(path, "Consumer.Echo", "hello", "world").endswith("hello world\n") + env = dict(os.environ, VOC_MODULE_PATH=str(directory)) + assert "run 1 kept 41" in run("Consumer.Run", env=env) + assert "libvoc-Absent-O2.so" in run(path, "Absent.Run", expected=1) + assert "not found" in run(path, "Consumer.Absent", expected=1) + assert "symbol file not found" in run(path, "Absent", expected=1) + assert "too long" in run(path, input="x" * 4096 + "\n", expected=1) + assert "needs a module path" in run("-P", expected=1) + + # Symbol reader handles the same rich interfaces as showdef. + root = Path(os.environ.get("VOCROOT", "/usr/share/voc")) + for name in ("Texts", "Files", "Heap", "Math", "oocStrings", "ulmObjects"): + interface = subprocess.check_output(["showdef", name], text=True) + expected = set(re.findall(r"^ PROCEDURE (\w+)\s*;", interface, re.MULTILINE)) + # Use a fresh name so even linked-in core modules exercise the .sym + # parser rather than falling back to their in-memory command registry. + symbols = (root / "2/sym" / f"{name}.sym").read_bytes() + (directory / "SymbolProbe.sym").write_bytes( + symbols.replace(name.encode() + b"\0", b"SymbolProbe\0", 1)) + actual = {line.strip().split(".", 1)[1] for line in run(path, "SymbolProbe").splitlines()} + assert actual == expected, (name, actual, expected) + + # Scan the entire installed symbol corpus, not only the example's types. + for symbols in (root / "2/sym").glob("*.sym"): + (directory / "SymbolProbe.sym").write_bytes(symbols.read_bytes().replace( + symbols.stem.encode() + b"\0", b"SymbolProbe\0", 1)) + run(path, "SymbolProbe") + + # Fail closed on unsupported versions, truncated streams and malformed tags. + good = (directory / "Consumer.sym").read_bytes().replace(b"Consumer\0", b"Broken\0", 1) + (directory / "Broken.sym").write_bytes(good) + assert "Broken.Run" in run(path, "Broken") + for content in (good[:1], good[:20], good[:-1], good[:1] + b"\x00" + good[2:], + good[:2] + b"\x80" * 40): + (directory / "Broken.sym").write_bytes(content) + assert "invalid or unsupported" in run(path, "Broken", expected=1) + + for name, descriptor in (("BadABI", "export const INT32 BadABI__voc_abi[8] = {1, 0};"), + ("NoABI", "")): + source = directory / f"{name}.c" + source.write_text('#include "SYSTEM.h"\n' + descriptor + + f'\nexport void *{name}__init(void) {{ return 0; }}\n') + subprocess.run([*shlex.split(os.environ.get("CC", "cc")), "-fPIC", "-shared", + f"-I{root / '2/include'}", str(source), "-o", + str(directory / f"libvoc-{name}-O2.so")], check=True) + assert "ABI" in run(path, f"{name}.Run", expected=1) + + dynamic = subprocess.check_output(["readelf", "-d", str(library)], text=True) + assert "libvoc-Provider-O2.so" in dynamic and "libvoc-O2.so" in dynamic + assert "$ORIGIN" in dynamic + interactive(directory) + print("vloksh tests passed: symbols, no-init completion, pty/history, shared dependencies, GC/state, errors, ABI") + + +if __name__ == "__main__": + temp_root = Path("/tmp/opencode") + temp_root.mkdir(exist_ok=True) + with tempfile.TemporaryDirectory(prefix="vloksh-test-", dir=temp_root) as directory: + main(Path(directory)) diff --git a/src/tools/vloksh/vloksh.Mod b/src/tools/vloksh/vloksh.Mod new file mode 100644 index 00000000..230741a9 --- /dev/null +++ b/src/tools/vloksh/vloksh.Mod @@ -0,0 +1,201 @@ +MODULE vloksh; + + (* Vishap Linux Oberon Kommand Shell. Commands execute in this process, + not in child processes, just as in loksh and the Oberon UI. *) + + IMPORT SYSTEM, Modules, SharedModules, Oberon, Texts, Out, Heap, Platform; + + VAR + prefix, modulePrefix: ARRAY 128 OF CHAR; + failed, quit: BOOLEAN; + + PROCEDURE -AInclude '#include "Terminal.h"'; + PROCEDURE -InitTerminal(complete: SharedModules.Visitor) + 'Vloksh_terminal_init(complete)'; + PROCEDURE -ReadLine(VAR line: ARRAY OF CHAR): SYSTEM.INT32 + 'Vloksh_read_line((char*)line, line__len)'; + PROCEDURE -AddCompletion(name: ARRAY OF CHAR) + 'Vloksh_add_completion((char*)name)'; + + PROCEDURE StartsWith(name, prefix: ARRAY OF CHAR): BOOLEAN; + VAR i: INTEGER; + BEGIN + i := 0; + WHILE (i < LEN(prefix)) & (prefix[i] # 0X) DO + IF (i >= LEN(name)) OR (name[i] # prefix[i]) THEN RETURN FALSE END; + INC(i) + END; + RETURN TRUE + END StartsWith; + + PROCEDURE Append(source: ARRAY OF CHAR; VAR target: ARRAY OF CHAR): BOOLEAN; + VAR i, j: INTEGER; + BEGIN + i := 0; WHILE (i < LEN(target)) & (target[i] # 0X) DO INC(i) END; + j := 0; + WHILE (j < LEN(source)) & (source[j] # 0X) DO + IF i >= LEN(target) - 1 THEN RETURN FALSE END; + target[i] := source[j]; INC(i); INC(j) + END; + target[i] := 0X; + RETURN TRUE + END Append; + + PROCEDURE CompleteModule(name: ARRAY OF CHAR); + VAR candidate: ARRAY 128 OF CHAR; ok: BOOLEAN; + BEGIN + IF StartsWith(name, prefix) THEN + COPY(name, candidate); ok := Append(".", candidate); + IF ok THEN AddCompletion(candidate) END + END + END CompleteModule; + + PROCEDURE CompleteCommand(name: ARRAY OF CHAR); + VAR candidate: ARRAY 128 OF CHAR; ok: BOOLEAN; + BEGIN + COPY(modulePrefix, candidate); + ok := Append(".", candidate) & Append(name, candidate); + IF ok & StartsWith(candidate, prefix) THEN AddCompletion(candidate) END + END CompleteCommand; + + PROCEDURE Complete(text: ARRAY OF CHAR); + VAR i, j: INTEGER; valid: BOOLEAN; + BEGIN + IF LEN(text) > LEN(prefix) THEN RETURN END; + COPY(text, prefix); + i := 0; WHILE (prefix[i] # 0X) & (prefix[i] # ".") DO INC(i) END; + IF prefix[i] = "." THEN + j := 0; WHILE j < i DO modulePrefix[j] := prefix[j]; INC(j) END; + modulePrefix[j] := 0X; + valid := SharedModules.EnumerateCommands(modulePrefix, CompleteCommand) + ELSE + IF StartsWith("help", prefix) THEN AddCompletion("help") END; + IF StartsWith("quit", prefix) THEN AddCompletion("quit") END; + IF StartsWith("exit", prefix) THEN AddCompletion("exit") END; + SharedModules.EnumerateModules(CompleteModule) + END + END Complete; + + PROCEDURE Help; + BEGIN + Out.String("vloksh - Vishap Linux Oberon Kommand Shell"); Out.Ln; + Out.String(" vloksh [-Ppath] [Module.Command [arguments]]"); Out.Ln; + Out.String(" vloksh < script run a batch of commands"); Out.Ln; + Out.String(" Module list its commands without initializing it"); Out.Ln; + Out.String(" Tab: completion; arrows: history; quit/exit/Ctrl-D: leave"); Out.Ln; + Out.String(" VOC_MODULE_PATH: colon-separated module directories"); Out.Ln; + Out.String(" Modules stay loaded; HALT or a fault terminates this prototype."); Out.Ln + END Help; + + PROCEDURE ShowCommand(name: ARRAY OF CHAR); + BEGIN + Out.String(" "); Out.String(modulePrefix); Out.Char("."); Out.String(name); Out.Ln + END ShowCommand; + + PROCEDURE Error; + BEGIN + Out.String("vloksh: "); Out.String(SharedModules.resMsg); Out.Ln; + failed := TRUE + END Error; + + PROCEDURE Execute(line: ARRAY OF CHAR); + VAR i, j, dot, start: INTEGER; name: ARRAY 128 OF CHAR; + command: ARRAY 64 OF CHAR; m: SharedModules.Module; + proc: SharedModules.Command; par, previous: Oberon.ParList; + W: Texts.Writer; + BEGIN + i := 0; WHILE (line[i] = " ") OR (line[i] = 09X) DO INC(i) END; + IF (line[i] = 0X) OR (line[i] = "#") THEN RETURN END; + j := 0; dot := -1; + WHILE (line[i] # 0X) & (line[i] > " ") DO + IF j >= LEN(name) - 1 THEN + SharedModules.resMsg := "command name too long"; Error; RETURN + END; + IF line[i] = "." THEN dot := j END; + name[j] := line[i]; INC(i); INC(j) + END; + name[j] := 0X; + IF (name = "quit") OR (name = "exit") THEN quit := TRUE; RETURN END; + IF (name = "help") OR (name = "--help") OR (name = "-h") THEN Help; RETURN END; + IF dot < 0 THEN + COPY(name, modulePrefix); + IF ~SharedModules.EnumerateCommands(name, ShowCommand) THEN Error END; + Out.Flush; RETURN + END; + name[dot] := 0X; + j := 0; start := dot + 1; + WHILE name[start] # 0X DO + IF j >= LEN(command) - 1 THEN + SharedModules.resMsg := "command name too long"; Error; RETURN + END; + command[j] := name[start]; INC(j); INC(start) + END; + command[j] := 0X; + WHILE (line[i] = " ") OR (line[i] = 09X) DO INC(i) END; + NEW(par); NEW(par.text); Texts.Open(par.text, ""); par.pos := 0; + Texts.OpenWriter(W); + WHILE line[i] # 0X DO Texts.Write(W, line[i]); INC(i) END; + Texts.Append(par.text, W.buf); + (* Set parameters before __init too, for modules that inspect them there. *) + previous := Oberon.Par; Oberon.Par := par; + m := SharedModules.ThisMod(name); + IF m # NIL THEN + proc := SharedModules.ThisCommand(m, command); + IF proc # NIL THEN proc ELSE Error END + ELSE Error + END; + Oberon.Par := previous; + Out.Flush; + Heap.GC(TRUE) + END Execute; + + PROCEDURE Run; + VAR i, status: INTEGER; line: ARRAY 4096 OF CHAR; + arg: ARRAY 4096 OF CHAR; ok: BOOLEAN; + BEGIN + i := 1; + IF i < Modules.ArgCount THEN + Modules.GetArg(i, arg); + IF (arg[0] = "-") & (arg[1] = "P") THEN + IF arg[2] = 0X THEN + INC(i); + IF i >= Modules.ArgCount THEN + Out.String("vloksh: -P needs a module path"); Out.Ln; Platform.Exit(1) + END; + Modules.GetArg(i, SharedModules.path) + ELSE + status := 0; + WHILE arg[status + 2] # 0X DO + SharedModules.path[status] := arg[status + 2]; INC(status) + END; + SharedModules.path[status] := 0X + END; + INC(i) + END + END; + IF i < Modules.ArgCount THEN + line := ""; + WHILE i < Modules.ArgCount DO + Modules.GetArg(i, arg); + ok := Append(arg, line); + IF i < Modules.ArgCount - 1 THEN ok := ok & Append(" ", line) END; + IF ~ok THEN Out.String("vloksh: command line too long"); Out.Ln; Platform.Exit(1) END; + INC(i) + END; + Execute(line) + ELSE + InitTerminal(Complete); + WHILE ~quit DO + status := SHORT(ReadLine(line)); + IF status = 0 THEN quit := TRUE + ELSIF status < 0 THEN + Out.String("vloksh: input line too long"); Out.Ln; failed := TRUE + ELSE Execute(line) + END + END + END; + IF failed THEN Heap.FINALL; Platform.Exit(1) END + END Run; + +BEGIN Run +END vloksh. diff --git a/src/tools/vloksh/voc-shared.py b/src/tools/vloksh/voc-shared.py new file mode 100644 index 00000000..5c7eedbf --- /dev/null +++ b/src/tools/vloksh/voc-shared.py @@ -0,0 +1,123 @@ +#!/usr/bin/env python3 +"""Build one VOC module per ELF shared object, with a single shared runtime.""" + +import argparse +import os +from pathlib import Path +import re +import shlex +import shutil +import subprocess +import sys + + +def installation(voc, model): + compiler = Path(shutil.which(voc) or voc).resolve() + prefix = compiler.parent.parent + roots = [prefix / "share/voc", prefix] + root = Path(os.environ["VOCROOT"]) if "VOCROOT" in os.environ else next( + (p for p in roots if (p / model / "include/SYSTEM.h").is_file()), roots[0] + ) + library = f"libvoc-O{model}.so" + directories = [root / "lib", prefix / "lib64", prefix / "lib"] + libdir = Path(os.environ["VOCLIBDIR"]) if "VOCLIBDIR" in os.environ else next( + (p for p in directories if (p / library).is_file()), directories[0] + ) + return root, libdir + + +def run(command, **kwargs): + print("+ " + shlex.join(str(part) for part in command), flush=True) + result = subprocess.run([str(part) for part in command], **kwargs) + if isinstance(result.stdout, str): + print(result.stdout, end="", flush=True) + result.check_returncode() + return result + + +def build(args): + root, libdir = installation(args.voc, args.model) + root = Path(args.root or root).resolve() + libdir = Path(args.libdir or libdir).resolve() + include = root / args.model / "include" + runtime = libdir / f"libvoc-O{args.model}.so" + if not (include / "SYSTEM.h").is_file() or not runtime.is_file(): + raise ValueError("cannot find VOC installation; use --root and --libdir") + + source = Path(args.source).resolve() + env = dict(os.environ, VOCROOT=str(root), VOCLIBDIR=str(libdir)) + directories = [Path.cwd(), *map(Path, args.library_path), libdir] + directories = list(dict.fromkeys(p.resolve() for p in directories)) + # -S translates only. Compile the generated C explicitly with PIC even if + # the installed compiler's default C command does not enable it. + result = run([args.voc, f"-O{args.model}", "-sFS", source], env=env, + stdout=subprocess.PIPE, text=True) + + # Let VOC read ASCII or binary V4 source and report the MODULE name; the + # generated __REGMOD must agree. Never assume it matches the file name. + declared = re.search(r"\bCompiling ([A-Za-z][A-Za-z0-9_]*)\.", result.stdout) + if not declared: + raise ValueError("cannot determine VOC's generated module name") + name = declared[1] + text = Path(f"{name}.c").read_text() + registered = re.search(r'__REGMOD\("([A-Za-z][A-Za-z0-9_]*)",', text) + if not registered or registered[1] != name: + raise ValueError(f"{name} is not a non-main module") + commands = re.findall(r'__REGCMD\("([A-Za-z][A-Za-z0-9_]*)",', text) + if len(name) > 19 or any(len(command) > 23 for command in commands): + raise ValueError("VOC's runtime registry limits module names to 19 and commands to 23 characters") + + symbols = subprocess.check_output(["nm", "-D", "--defined-only", str(runtime)], text=True) + runtime_symbols = {line.split()[-1] for line in symbols.splitlines() if line.split()} + imports = list(dict.fromkeys(re.findall(r"__MODULE_IMPORT\((\w+)\)", text))) + dependencies = [] + for imported in imports: + library = f"libvoc-{imported}-O{args.model}.so" + if f"{imported}__init" in runtime_symbols: + # Never duplicate modules already provided by the shared runtime. + continue + if not any((directory / library).is_file() for directory in directories): + raise ValueError(f"missing shared dependency {library}; build {imported} first or use -L") + dependencies.append(f"-l:{library}") + + shared = Path(f"{name}.shared.c") + shared.write_text(text + f'''\n/* Shared-module ABI shape, checked before calling __init. */ +#include "Heap.h" +export const INT32 {name}__voc_abi[] = {{ + 1, sizeof(ADDRESS), sizeof(SHORTINT), sizeof(INTEGER), sizeof(LONGINT), + sizeof(SET), sizeof(Heap_ModuleDesc), sizeof(Heap_CmdDesc) +}}; +''') + library = f"libvoc-{name}-O{args.model}.so" + cc = shlex.split(args.cc) + includes = [Path.cwd(), include, *map(Path, args.include)] + flags = shlex.split(os.environ.get("CFLAGS", "")) + ldflags = shlex.split(os.environ.get("LDFLAGS", "")) + extra_libs = shlex.split(os.environ.get("LDLIBS", "")) + run([*cc, *flags, "-fPIC", "-Wno-stringop-overflow", "-shared", shared, + *[f"-I{directory}" for directory in includes], *ldflags, + *[f"-L{directory}" for directory in directories], + "-Wl,-z,defs", f"-Wl,-soname,{library}", "-Wl,-rpath,$ORIGIN", + "-Wl,--no-as-needed", *dependencies, f"-lvoc-O{args.model}", + *extra_libs, "-o", library]) + print(f"Built {library}; keep {name}.sym and {name}.h for development/completion.") + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("source", help="one non-main Oberon module") + parser.add_argument("--model", choices=("2", "C"), default="2") + parser.add_argument("--voc", default=os.environ.get("VOC", "voc")) + parser.add_argument("--cc", default=os.environ.get("CC", "cc")) + parser.add_argument("--root", help="VOC resource directory, e.g. /usr/share/voc") + parser.add_argument("--libdir", help="directory containing libvoc-O2.so") + parser.add_argument("-I", "--include", action="append", default=[]) + parser.add_argument("-L", "--library-path", action="append", default=[]) + try: + build(parser.parse_args()) + except (ValueError, OSError, subprocess.CalledProcessError) as error: + parser.exit(1, f"voc-shared: {error}\n") + + +if __name__ == "__main__": + main()