starting work on dynamically loaded modules and shell

doc/SharedModules.md

@@ -0,0 +1,158 @@
+# Shared modules and vloksh (Unix prototype)
+
+`vloksh` is a minimal VOC counterpart of polpo's loksh. It executes Oberon
+`Module.Command` procedures in its own address space, loading native ELF `.so`
+libraries on demand. The shell and loader are optional: the normal compiler
+build and the installed runtime are unchanged.
+
+## Try it
+
+Requires an installed VOC with its shared runtime and development headers,
+a C compiler, GNU Readline development files, Python 3 and binutils.
+The current shell uses the default Oberon-2 type model (`-O2`).
+
+```sh
+make vloksh
+make vloksh-test
+make vloksh-demo
+build/vloksh/vloksh -Pbuild/vloksh/strutils-demo
+```
+
+The demo reads sources from `../strutils` without modifying that repository.
+Override its location with `STRUTILS=/path/to/strutils`.
+Inside the shell:
+
+```text
+> StrDemo
+  StrDemo.Echo
+  StrDemo.Run
+> StrDemo.Run
+shared
+modules
+really
+work
+calls: 1
+> StrDemo.Run
+shared
+modules
+really
+work
+calls: 2
+> StrDemo.Echo hello world
+hello world
+> quit
+```
+
+Tab completes module names, then their commands. Readline supplies editing,
+session history and filename completion for arguments. Module and command
+discovery does **not** load a library or run a module initializer. It reads the
+existing `.sym` files, selecting exported, parameterless proper procedures,
+as showdef's `OPT.Import` reader would. Already initialized modules are also
+discoverable through VOC's command registry.
+
+The shell also supports single-command and batch operation:
+
+```sh
+build/vloksh/vloksh -Pbuild/vloksh/strutils-demo StrDemo.Run
+printf 'StrDemo.Run\nStrDemo.Run\n' | build/vloksh/vloksh -Pbuild/vloksh/strutils-demo
+```
+
+`-Ppath` or `-P path` overrides `VOC_MODULE_PATH`. Paths are colon-separated;
+by default the loader searches the current directory and the directory of the
+actually loaded `libvoc-O2.so`. Symbols are searched in those directories,
+then `VOC_SYM_PATH`, or the installation's `2/sym` directory when it is unset.
+`VOCROOT` can override the installation root for symbol discovery.
+
+Build overrides for another installation:
+
+```sh
+make vloksh VOC=/opt/voc/bin/voc VOCROOT=/opt/voc VOCLIBDIR=/opt/voc/lib
+```
+
+Build products stay in `build/vloksh`. Nothing is installed by these targets.
+
+## Building and packaging modules
+
+The experimental helper builds **one module per shared object**, in import
+order, in the current build directory:
+
+```sh
+python3 /path/to/compiler/src/tools/vloksh/voc-shared.py /path/to/strTypes.Mod
+python3 /path/to/compiler/src/tools/vloksh/voc-shared.py /path/to/strUtils.Mod
+python3 /path/to/compiler/src/tools/vloksh/voc-shared.py /path/to/StrDemo.Mod
+```
+
+For module `M` it produces `libvoc-M-O2.so`, plus the usual `M.sym`, `M.h` and
+generated C. It translates with VOC and compiles explicitly with `-fPIC`.
+Every external import becomes an ELF `DT_NEEDED` dependency; modules already
+provided by `libvoc-O2.so` are not duplicated. `$ORIGIN` run paths let libraries
+in the same directory find each other. Imports in another directory need a
+system linker search path or `LD_LIBRARY_PATH` at runtime; `-P` locates the
+requested module but cannot override the ELF linker's dependency search.
+
+The helper supports `--root`, `--libdir`, `--voc`, `--cc`, `--model C`, `-I`
+(additional C includes) and `-L` (shared dependencies). `OBERON`/`MODULES`
+remain VOC's symbol search configuration. It honors `CC`, `CFLAGS`,
+`LDFLAGS`, `LDLIBS`, `VOCROOT` and `VOCLIBDIR`.
+
+A Gentoo package can install:
+
+| File | Location / purpose |
+| --- | --- |
+| `libvoc-strTypes-O2.so`, `libvoc-strUtils-O2.so` | `/usr/$(get_libdir)`; runtime libraries |
+| `strTypes.sym`, `strUtils.sym` | `/usr/share/voc/2/sym`; compilation and shell completion |
+| `strTypes.h`, `strUtils.h` | `/usr/share/voc/2/include`; C backend compilation |
+
+Static archives can coexist for existing consumers; `.sym` and `.h` are still
+needed when compiling against a packaged module. Runtime execution itself
+does not need symbols, but completion of an unloaded module does. No command
+index or other completion metadata is required. The overlay ebuilds have not
+been changed by this experiment.
+
+## Loader contract and limitations
+
+- `SharedModules.ThisMod` opens `libvoc-M-O2.so` with
+  `dlopen(RTLD_NOW | RTLD_GLOBAL)`, checks a small ABI descriptor and calls
+  `M__init`. That initializer registers the module, its commands and GC roots
+  in the existing shared VOC runtime. `ThisCommand` uses the existing registry.
+  Existing `Modules.ThisMod` is unchanged; clients needing on-demand lookup
+  use `SharedModules.ThisMod` in this prototype.
+- Handles and pointer-sized values use `SYSTEM.ADDRESS`, not `LONGINT`.
+- The helper embeds `M__voc_abi`, checking pointer/basic-type sizes and runtime
+  descriptor sizes. This is a **shape check, not a complete ABI/versioning or
+  imported-interface fingerprint check**. Use matching compiler/runtime,
+  architecture, type model and dependency interfaces. Unrelated Ofront `.so`
+  files are not compatible. Dependencies must be built with the same contract.
+- The shell and all loaded modules must use **one shared** `libvoc-O2.so`.
+  Statically embedding separate runtime/heap copies in plugins is unsafe.
+- Libraries remain loaded until process exit. The loader pins a directly loaded
+  module's registry entry so `Modules.Free` cannot remove its GC roots. Imported
+  modules remain referenced by their clients. There is no unload/reload yet:
+  VOC's static type descriptors, pointer enumerators and finalizers can outlive
+  a command, making a casual `dlclose` unsafe.
+- Commands take no formal parameters or result; arguments are available as
+  `Oberon.Par.text` starting at `Oberon.Par.pos`. General exported functions
+  remain callable by importing modules, but are not shell commands.
+- HALT, assertions and faults terminate the prototype; there is no loksh-style
+  trap recovery, process isolation, Unix pipeline syntax or job control yet.
+  Load only trusted native modules: they have the shell's full permissions.
+- The existing registry limits module names to 19 and command names to 23
+  characters. The helper rejects names that would be truncated.
+
+## Layout and later V4 work
+
+`src/library/v4/SharedModules.Mod` is the reusable, UI-independent loader;
+`CommandSymbols.Mod` is a read-only `.sym` command reader following
+`src/compiler/OPM.Mod` and `OPT.Mod` (format `F7 84`). Unsupported or malformed
+symbols return no partial completions. The small C support layer handles
+POSIX library/path operations. The shell and Readline bridge live separately
+in `src/tools/vloksh`.
+
+A later graphical port fits under `src/library/v4/ui`, rather than mixing
+UI dependencies into the existing headless library. Its entry command could
+be launched by the same loader. However, VOC already supplies headless
+`Texts` and `Oberon` modules: a loaded UI cannot simply replace those modules
+with incompatible records/interfaces of the same names. The port must first
+unify the common interfaces or use distinct CLI/UI modules and import aliases,
+as polpo does. X11 bindings, fonts/text resources, event loop and trap handling
+also need adaptation. This prototype establishes loading, not that UI port.