word: compile call jmp execute call jmp addr label asmtime runtime num: write 00 push 00 postpone: call jmp call jmp label asmtime runtime 00 00
tl;dr - starting from nothing but the provided colors, quickly dsl yourself artisanal output using an unsafe but fun concatenative DIY assembler.
It is true, in Blue you start with nothing and quickly dsl yourself artisanal
output. But how? Is it hard? At first, very likely. Once it clicks though its
really pretty easy. This guide assumes some knowledge of forths and x86_64
assembly on Linux. If you need a primer there are several links at the bottom
of the repo’s README.md.
To begin to describe how Blue works, lets first look at what is likely the most minimal Blue program that emits a valid executable:
Minimal impl of `/bin/true`. Since green numbers are written directly to the output, this writes a hardcoded elf header and machine code to issue the exit system call. While valid, typical Blue programs will build up a vocabulary across multiple `.bo` files. However it does all start with green numbers.
03010102464C457F 0000000000000000 00000001003E0002 0000000000400078 0000000000000040 0000000000000000 0038004000000000 0000000000400001 0000000700000001 0000000000000000 0000000000400000 0000000000400000 000000000000007C 000000000000007C 0000000000001000 050F3CB0[top]
$ blue < x86_64/Linux/cmd/true/true.b > bin/true && chmod +x bin/true
$ ./bin/true && echo $?
0
$ stat -c "%s %n" bin/true
124 bin/true
We can see that true.b yields a 124 byte binary that when run exits with a
status code of 0. Each of the 248 [0-9A-F] are nibbles, so each pair is a byte
in hex form. In Blue green numbers are written directly to the output, much
like a db or dq that an assembler may provide. So here we write a minimal
120 byte ELF Header followed by four bytes to issue the exit system call:
0x050F3CB0 which disasm’s to:
mov al, 0x3C
syscall
Some problems: the ELF Header’s entry is hardcoded to 0x00400078 and the on
disk and in memory sizes are 0x7C. Hand typing machine code as green numbers
tends to get old. Lastly, the machine code here really assumes that it is the
first thing to run - if anything above prevents edi or eax from being
zero’d out this will no longer work.
In short, true.b is valid but not how we’d like to code in Blue. We will
chalk this up to a learning exercise and try again, this time building up a
vocabulary from green numbers as we go.
The shadow block for true.b says that “it does all start with green numbers”.
When using Blue any code written to dst can be called at asmtime. This can
be used to build up a dsl to describe your program. As an example, the first
block in many Blue programs is:
`basis` defines the words needed to begin encoding x86_64 machine code. After this file we escape the need to hand jit words with only green numbers. The machine code here is tightly coupled to the registers used by Blue v7.
; C3 C3 >ds 49 0089 08C08349 ; ds>2 08E88349 49 088B ds> 08E88349 49 008B ; | ds>2 48 C809 >ds << ds>2 48 E0D3 >ds ,1 ds> AA ;[top]
Here we are starting to build up our vocabulary. Since Blue ships with no
builtin words, we start with ; (semi) then >ds (push to data stack), etc.
Semi starts with a postponed 0xC3 meaning that when the word is called we
want 0xC3 to be written to dst. Then an immediate 0xC3 follows, so that
when semi is called it will return to the caller. Right before ds>2 you can
see semi being called as a blue word, which executes at asmtime. Later the
defintion of | (or) begins by popping two off the data stack, some machine
code to or rax, rcx then push rax to the data stack. Now that these words are
in place, we can start to encode x86_64 instructions without only green
numbers:
rex.b 41 ; rex.w 48 ; rex.wb 49 ; rex.rw 4C ; /0 C0 | /r 03 << | /0, /0 ,1 /r, /r ,1 /r0, /r /0 ,1 add 01 /r0, sub 29 /r0, xor 31 /r0, mov 89 /r0, mov# B8 | ,1 []mov 89 /r, []mov1 88 /r, push 50 | ,1 pop 58 | ,1 stosq rex.w stosd AB ; stosw AB66 ; stosb AA ; syscall 050F ; axchg 90 | ,1[top]
Blue numbers are pushed to the data stack instead of written to dst. In the
case of push for example, we don’t want 0x50 on the data stack until the
word is called, so we postpone it. Another interesting aspect of push is that
it ends with a jmp to ,1 which is like a tail call. You could issue a
regular call to ,1 then have a semi, but that is just an extra byte in the
output.
Now that we have some encodings defined, typically we’d define the registers that we will be using:
a ds rax/r8 00 ; b rcx 01 ; c srce rdx/r10 02 ; last rbx 03 ; dstb r12 04 ; ops r13 05 ; src rsi 06 ; dst rdi 07 ;[top]
In the bytecode, bytes
0x00to0x7Fare ignored by the interpreter and by default rendered as black text. This allows for, but is not limited to, adding comments to the bytecode.
At this point we can really start to see why Blue is referred to as a
concatenative DIY assembler. Next we define some helper macros that when
called as blue words will (by calling our other previously defined macros),
pop values off the data stack and write the machine code to dst that will
set some registers at runtime:
ds! rex.b ds mov# ,4 dst! 8B 253C ,4 dstb! rex.wb dstb swap mov last! 8B 251C ,4 ops! rex.b ops mov# ,4 srce! rex.wb srce swap mov[top]
Brick by brick your program is being described, right down to how the encoding
is performed. Some interesting things to note: Blue is really just like a
colorForth based assembler that has no separate macro syntax. Since it makes
no assumption about the output format (and has no builtin words) things like
the dozens of mov variants are not
included. Instead you simply define the ones you need.
Using Blue to emit x86_64 machine code is just one use actually. There is no reason you couldn’t dsl yourself a means to describe a bitmap, markdown file, etc.
To wrap up, lets fast forward and take a look at what the tail end of a Blue program may look like after all the bricks have been laid:
$* entry ! init srcsz _src stdin read a dup src add srce! _dst dst! dst dstb! _dict last! org ds! opa ops! 00 next srce? ,num $ $^ - $^ stdout write bye then src1>a char? next then >ds dup dup F7 & swap 07 & ds>3 src>a callop next[top]
For the sake of brevity, not all words that are used above have their definitions in this article. Magenta words take the address of a word and pushes either its asmtime or runtime address onto the data stack.
The above example is the entry point of a program that counts the number of red words in a given bytecode stream. Its full source can viewed here.
Scroll up and back down here again. In short time we went from only writting machine code via green numbers to writing our program via a high level dsl - a dsl that was formed by asmtime execution of all the previous words we defined. Words for us, by us (and our program naturally).
Once the vocabulary has been built, it no longer looks like we are inscribing
x86_64 machine code. In exchange for this little bit of effort you get
complete control over your output (and how you describe it). Hopefully this
helps illustrate how Blue works and how you can go from nothing but the
provided colors to artisanal output in short order.