Jim's Depository

this code is not yet written
 

Femtoblogger gains dark mode support

If you are thinking things changed, you are probably getting the dark mode support. It's your browser's choice. I send the same stuff to you, but the CSS is now littered with @media (prefers-color-scheme: dark) { sections to change things up for dark mode.

I wish Safari didn't put all the editable text, radio buttons, and buttons in blinding white. I wonder if I gorked something up there.

Update: Yeah. It was me. Somewhere in the process of switching to use CSS variables for my colors the problem went away. I probably had a background forced to something unfortunate.

Femtoblogger News

I've switched from the Ink markdown processor to Down. I liked that Ink was pure swift, but I kept spending time enhancing it to meet my needs. Down uses libcmark, which I'm not wild about having inside my server, but hopefully I won't spend any more time thinking about it.

It's a testament to both that swapping took just a couple minutes. Half of it finding the typo in my Packages.swift file.

print("Hello World! I'm beautiful!")

As long as I was fiddling with femtoblogger I touched up the CSS so code will stand out a little better, and then splurged and added Prism to highlight source code for you. I feel a little bad about the 12k of javascript, but you only load it once. We will survive this.

So far I recommend Prism. It was a CSS add in <HEAD> and a <SCRIPT> add by </BODY> and everything worked.

The biggest add to femtoblogger is an out of band management interface, but you don't get to see that.

Thoughts on Swift Argument Parser

I'm a big fan of Swift Argument Parser. It generally makes command line argument processing painless.

But…

That assumes you are in its primary use case as the main program's command processor.

What if you want to use it to drive a CLI?

You no longer have a primary command, you'll have to sort that out from the first argument. You could do that outside with a dictionary, or make a single fake command and make the rest subcommands. This has some frustrating shortcomings. It would be nicer to add a layer for "one of many commands".

What if your command needs context?

When my SSH command executer in my server wants to run a command, it needs to know how to write its output back to the remote SSH client. The .run() methods of ArgumentParser don't take arguments. The commands themselves are structs and really don't want you to be clever and make them classes.

What if the shell didn't break my command into an array of string?

I weep each time I see someone break a command with .split(separator:" "). I mean, at least remember to set the omittingEmptySubsequences that you might not know exists. But what about quoted strings? What about backslashes? What if I want one of the arguments to be an empty string?

Problem Solved

I built ParsableCommands which adds a "one of many commands" layer to ArgumentParser and includes a proper string tokenizer if you need it. Easy to use and very little code.

The context issue is a little harder. The are an astonishingly large number of ways to fail at this in Swift. I tried things for about 6 hours before settling on a per-context-type derived protocol of ParsableCommand, adding an new .run(context:MyType) method to ParsableCommand, and using a switch to look at the command between parsing and running and figure out which kind of context it needs from its type.

This doesn't lend itself to a package solution, but the example program in the ParsableCommands package shows how to do it.

Synergy

Combining this with the SSHConsole works very well for having an out of band diagnostic and management interface to an HTTP server.

Add an SSH Console to you Swift Server

I have a number of servers written in Swift. Mostly these are HTTP backends. As anyone who has written backends knows, you always want to query some information out of them or maybe change a setting. Then ensues a carving off if the URL space, writing a bunch of command handlers, fretting over how to keep users out of them, and a creeping need to fuss with the HTML.

Earlier this week I got nerd sniped by the Swift NIO group when they released SwiftNIO SSH and problem solved!

I built SSHConsole to trivially add an SSH listening port to your existing server and let you build your management commands as plain text generating commands.

Thoughts on NIOSSH

  • There was a fair bit of frustration getting public key authentication working, but mostly because it was too easy. Some more documentation would help here, but you can just look at my example Echo program and the library and see how it all works.
  • The private key datatype is overly opaque. It makes you rewrite part of it so you can serialize and retrieve your SSH host key.

Thoughts on the Rest of the Project

  • I actually ended up wasted a lot more time struggling with Swift Argument Parser since I was outside its normal use case. There's another post on that.
  • As usual, dressing up for GitHub and adding the inline documentation took about as long as the design and coding. And I left out the high level documents because I was tired at the end. You'll have to live with the examples.

Adding an SSH management console to a Swift server

I notices NIOSSH today for SSH support in Swift. I'm going to try it as a management console for one of my HTTP backends. For an idea of how much effort is involved I'll give a running tally here.

T+0

  • Go look at https://github.com/apple/swift-nio-ssh. Read a lot of it.

  • Discover that there is already an example server hiding in Sources/NIOSSHServer

  • Copy it in, drop out the code for port forwarding

T+1hr

  • I can connect to my server, and it crashes. Progress.

  • Realize the sample server is trying to use /usr/local/bin/bash which I don't have.

  • Get rid of all the pipe code since I'm not going to be interactive, at least yet. Introduce a bunch of bugs. Get lost in event loops, futures, and promises and the mysteries of unwrapInboundIn and friends.`

T+2hr

  • Success! I have built a really complicated echo server.

T+4hr

  • Much better understanding of NIO ChannelHandler, threw out about half the previous code.

  • Got in a fight with Swift over how to say a parameter conforms to a protocol. Ended up with an ugly generic solution. Not happy.

  • Boiled down to a SSHCommandHandler class which is used as a channel handler. It gets a command and dispatches it to a doCommand(command:to:environment:) function which you override.

  • The Protocol fight was because I just want that to: parameter to be a TextOutputStream. I don't want to tell you about my specific version. That collides with the inout nature of the print(_:to:inout TextOutputStream) call.

T+6hr

Mostly liking the shape of the code. In your main or somewhere likely you will…

let console = SSHConsole(passwordDelegate: Authenticator())
try console.listen()

…later you can…

try? console.stop().wait()`

T+10hr

And just like that I hit a wall. Swift Argument Parser does not have a mechanism for you to pass in state as you dispatch the command. There is a run() with no arguments and that's it. I spent 4 hours exploring the vast array of things which don't work. I landed on a few that do, but they are a bit gross.

Here's what I ended up with:

  • Define a struct for my context.

  • Extend the ParsableCommand to require a context variable of my type. I called it ConsoleCommand

  • I define my commands with structs conforming to ConsoleCommand. Each one has to have a var context:ConsoleContext? = nil in it.

  • Now my run() can look at its context and everything is happy

  • Except… ParsableCommand conforms to Decodable and an Optional<Foo> only conforms if Foo conforms, and my context object can't conform because there is no sane null value for an output stream!

  • The textbook solution is to specify my own Codable keys and leave out the trouble maker, except these objects are full of variables because they have entries for all the command syntax elements.

  • … hours are wasted …

  • I finally implemented a @transient property wrapper for nil-able values which sets them to nil on a decode and doesn't write on an encode. This is inspired by a feature rejected by the Swift designers.

  • I don't like it. I resent having to put a @transient var context:ConsoleContext?=nil in every command, but when I tried using classes instead of structs for the commands it didn't parse, so that may not be supported, so no inheriting my boilerplate.

  • I may go back and make my context be Decodable even if I have to make some insane values to populate it. Then I can lose the property wrapper and also save some unwrapping.

All in all a typical day of programming. Lot's of good progress then I step in a bear trap and have to chew my leg off before continuing.

T+20 Hours

All wrapped up and published on GitHub with API level documentation and very little high level documentation. Just the way I hate to use software. You can find it at SSHConsole

I've Overranted

I was typing up my thoughts on using SwiftUI and broke femtoblogger! My screed gets truncated about half way through.

I'm not sure if that means I should:

  • settle down?
  • fix femtoblogger?
  • tighten up my writing?

Who am I kidding… where did I leave that source code…

Update: I try to be a good modern programmer and haul in functionality from strangers on GitHub instead of spending weeks writing it myself… and then… BAM. The Markdown to HTML translator I'm using would allow raw HTML through, which in a public environment is irresponsible, so I kill any HTML I find. Sadly, if you include <table> in your comment, the markdown engine thinks everything from the opening tag to the end is the HTML and I kill it.

I guess I'll have to add a real "safe mode" to the markdown translator. The author hasn't integrated a pull request in months, that is kind of deterrent to making nice additions.

Update: Three hours to study the markdown processor enough to figure out how to cleanly add a safe mode and tests. I'd have been done faster until I realized it was written by John Sundell. Google frequently brings me his articles for solving Swift conundrums. He deserves an extra tidy patch.

First thoughts on SwiftUI

History

About 20 years ago Apple rolled out AppKit, the NS* world of user interface programming, though it was already about 10 years old.

About 13 years ago Apple rolled out UIKit, a vaguely similar but significantly altered child of AppKit for the iPhone.

Over the years those frameworks morphed slightly under the coding fashions of the time. Extension by subclassing gave up some ground to extension by delegation, layout went from "chains and springs" to a constraint solver system, the primary client language changed from Objective C to Swift. Warts and debris accumulated in the APIs and backwards compatibility kept them forever.

In recent years, Apple has stopped maintaining and producing the high level documentation which describes how to use a subsystem. They still have API level documentation which describes each individual microscopic fragment, but they no longer produce the fabulous high level documents which tell you how to use an API to solve problems, and show you which areas you should be using, and which are leftovers.

The result is that in 2020 I pity anyone learning AppKit or UIKit. The web is filled with people asking questions and getting answers of varying quality, frequently low, which then become silently obsolete.

Present

18 months ago Apple released SwiftUI. This is a clean start. Not having any pressing new UI needs I decided to let it stew for a year before looking at it.

In a nutshell SwiftUI eschews graphical or imperative code creation of user interface for a terse "in source" declarative approach. Here, terse is largely accomplished with a few of language tweaks that let return be omitted at the end of a function and let let x = X(); x.add(A); x.add(B); x.add(C); return x be written as X { A B C }. It's a lot nicer to read. Writing is weird until you read how function builders work about 12 times, but you can copy-paste your way into an interface even if you don't understand why. Remember: a View is just a regular struct until you get into the body initializer then you are concatenating Views without commas between them

What I Built

I built a cross platform iOS/macOS app with a widget to display some telemetry of remote battery voltages, weather conditions, and a camera picture. It took me about two days to learn enough SwiftUI to make the simple non-interactive widgets and display screens. A fair bit of that was working out WidgetKit which isn't hard, but it took me a while to realize that because my data was driven by realtime data I could just chuck most of the complexity and not hurt my energy consumption story.

The Good

  • The user interface is nicely legible in the source files. It says what it does and does what it says.
  • Making a Widget, with its inherently simple display is really easy. (maybe)
  • Cross platform is beautiful! I essentially paid no attention to it and things just worked. I never once wanted to fly to Cupertino and find the people that think we don't want something to abstract UIColor and NSColor, and UIImage and NSImage for us poor cross platform app writers.

The Bad

  • Those syntax shenanigans SwiftUI uses to look really clean? They really hurt the compiler and Xcode's ability to produce useful error messages and code inspection information (like throwing down the template arguments as you type a function call). I lost track of how many times Xcode threw up its little hands and asked me to file a bug report while I typed. Lesson: Move everything except the view layout itself to a different file.
  • It looks simple, because it is simple. In many ways SwiftUI is in that early project phase where they just punt on the hard stuff and tell you to live with the simple way, it's good enough. And if your app's visual can be "good enough" or rejiggered into something that SwiftUI can do, then good for you.
  • You may not be able to reach excellence. I draw strip charts of time series data, they look a lot like they come out of an old pen strip recorder. They have horizontal and vertical scale lines which are generally as thin as possible on the display. This requires that the lines be aligned to the display pixel grid or some of them become fuzzy from being smeared over two rows of pixels. You don't notice this with thick lines, but hairlines need to handle this. As near as I can tell, there is no way to find out my View's position in device coordinates to then make my drawing come out right.
  • Not all Views are suitable for deployment to customers. On the Mac, the HSplit appears to have a 1 pixel wide hit zone for moving the divider and no visual indication that the split is moveable. The TextEditor leaves behind visual artifacts if you start selecting text and drag up and down. These are problems that I can't fix without source code or just writing a replacement.
  • You can draw, sort of, there is a Path view that draws a path, and you can absolutely place Text, but only by their center point. I rebuilt my stripchart view, but it works by computing a Path for each line width, color combination and drawing them in a View, then an array of Text with absolute placements to put the labels on. It's gross. Maybe I should have just wrapped my UIKit stripchart and gone on, but this works cross platform.
  • If you do not embrace Combine, you will likely have a lot of parameters being passed up and down your View trees.
  • Swift's inability for let members of a struct to reference each other really hurts when you have related data. I get it, there are reasons, but I'm pretty sure a topological sort in the compiler would take care of that. I dealt with it by making new objects which encompass all the other objects and taking care of the relationships in that init.
  • There are things which will break you. Consider the Label() type. It easily lets you pair an icon and a string. Just the thing for lots of lists. You see them everywhere. Now try to use one outside a list, because say you are a postage stamp sized widget that doesn't have interaction. Suddenly your VStack of Labels looks like garbage because the text no longer lines up and the icons aren't centered over each other. Not illegible, just sloppy. Can you fix it? There is no high level documentation from Apple, so it's off to the internet where possibly with a combination of Geometry and AlignementGuides (which are a little pile of hack) you can probably get the text in line and the icons to right align, which is better than nothing. About 50 lines of obtuse code.
  • No problem, don't use VStack{ Label() }, use a Table. They didn't make a table. I really can't fathom why unless someone was so scarred from the HTML <table> wars or intimidated by the "displays 100000 rows just fine" versions in the older APIs. Developers want to be able to have a few columns and a dozen rows to organize our displays and data. Give us some column sizing, and alignment options (including radix point, we have needs) and lets us get on with life. I don't want to use a native table, I'm trying to be cross platform here, remember "We love the Mac." (As quoted by Tim Cook twice a year at WWDC and Mac product release.)

So What Does a Grumpy Old Coder Do?

So, after all that frustration, the next day I started a new Mac app and chose SwiftUI because it really is easy and seductive and this app is mostly just for me to analyzer data and maybe a few other people. I'm not answering to a graphic designer with a Photoshop Phantasy user interface. It has to be nice, but if SwiftUI doesn't want to go somewhere, I have the liberty to not go there.

I'm displaying a couple of side by side files, source code and LLVM intermediate language generated from it. Compile the source, look at the LLVM and generated assembly code. Maybe some parse trees or internal AST dumps.

  • The App prototyping was glorious! Quick and easy to noodle about with.
  • @ObserableObject worked nicely for tying my internal data to the screen. Things Just Worked! Hit the compile button, replace the LLVM text, and the screen updates.
  • TextEditor fell on its face. Rendering bugs aside, it doesn't have the capabilities to do what I needed and no where to hook in and extend. SwiftUI does the easy stuff™
  • I wrapped an NSTextView, so goodbye any pretense of multiplatform, but that was never in the cards since its tied to a compiler anyway. I wasted about half a day learning to wrap and coming to grips with the Coordinator and where to keep my delegate and then that I really kind of needed to use Combine to sink my updates around, but after those hard lessons were learned it was pretty good.
  • Totally not SwiftUI, but if I click on a line in the source and I want to jump to the LLVM assembly that corresponds… I just feel I shouldn't have had to fight NSTextView quite so hard. I won in the end, but it took hours. My text is NSAttributedString type so it can have color and line spacing and whatnot, that makes everything like walking through mud.
  • I wanted little line numbers on the side of the NSTextView. Your API is older than your developers! This has to have come up. Fortunately you can find a solution in GitHub from Yichi Zhang. I was ready to waste a few more hours before declaring it a mess and giving up and his one file ruler class worked the first time I hooked it in! (I still had work on it a bit to use small, digit spaced text and to move the digits down to the baseline of the text. That's a little more LayoutManager and NSGlyph than most people want to know about.)
  • I got in a fight with the ToolBar. I just wanted my compile icon with the word "compile" under it because, really, when I said "compile icon" you visualized absolutely nothing. There is no picture for "compile". This is, sadly, too hard. The various toolbars will draw my Label() in different ways, none like that, except the style meant for Preferences which you can not specify in SwiftUI for some reason. It's all hard to work out because although all the API fragments are documented, there are no high level documents, so how to put them together and where to specify them is left as an exercise to the internet Oijia board. I could force it by making the toolbar item be a VStack, but then it renders poorly. After 90 minutes I gave up and make a Button with the text and symbol in it.
  • Adding a menu and menu items would have take 30 fewer minutes if Apple wrote the high level documentation.
  • I've harped on high level documentation, think about putting a NSTextView into an NSScrollView so it can scroll, probably a fair few bits to get right there. You can google it and get a bunch of answers, most either miss something important or do a lot of unnecessary stuff. But how about this: Putting an NSTextView Object in an NSScrollView A small amount of clean code, from people who know how the underlying parts work and can see their source code to verify assumptions. Look at that table of contents, it covers everything to get you up and going. Then you can work with any fiddly bits you need to customize for your situation. These documents require a team of engineers who can write and are expensive, but the number of man hours they save your customers is immense.
  • So at the end of the first day I had listings in parallel columns with syntax coloring. By the end of the second day I had clicking on source jumping to the appropriate spot in the LLVM, line numbers, assembly listings (goodbye sandbox, can't invoke clang from inside the sandbox without heroics). It's now the third day and although I still have a little list of features to add or try and reject, I'm writing this posting instead because my app is ready to use.

Conclusion

  • Use SwiftUI for Widgets. It's really your only choice there. If you already have code to gather your data, budget two days to make your first widget.
  • Use SwiftUI for quick-and-dirty internal apps where user interface defects aren't show stoppers.
  • Maybe use SwiftUI for the high level of a complicated app? Be ready to embrace Combine. I worry about Combine creating unfathomable interactions, but I haven't used it on a complicated enough project to know.
  • If you have anything complicated… table, styled text, drawings, be ready to drop into UIKit or AppKit.

Recommendations to Apple

  • Print out a banner that says "They do need that!" and hang it in the engineers' room (if we ever get to use rooms again).
  • Bring on a couple engineers and writers to write the high level documentation. Also get someone to bonk the coders on the head when I lookup API documentation for foo(bar:Int) and the detail page tells me that it is foo with an argument named bar of type Int which foos. I kind of was looking for a little more than that even if it seemed obvious to the coder when he hit add documentation in Xcode.
  • Send someone out across the internet for postings on "how to" and "swifui" and document all of the API usage questions people have in that high level documentation.

New content for the first time in 5 years has attracted comment spam.

The robots are reading at least. The rate is low enough I'll just manually delete them for now.

I used to use the google recaptcha, but I know I'm annoyed at finding stop lights and street signs and in a more tracking sensitive era I don't feel like feeding people's reading habits to third parties.

That probably means I'll eventually have to make my own human detector system and lose a week to that.

Back to the Language

This blog started 13 years ago musing about a language. I'm back to the language.

The goal is a language to use for the user mode code of a concurrent communications based operating system. I need it to catchiest mistakes at compile time, be agile enough to try language experiments, and have reasonable performance which is maybe a small predictable integer off of top notch C.

The State of the Language

  • I chose a name! Yes, this usually happens late for me, but it is atuin. After the turtle.
  • The compiler written in Swift, which is by far the most productive language I've used for writing a compiler. It emits LLVM IR (in text), which clang then compiles into an executable. I've gone to some length to keep my LLVM IR legible, it really helps diagnosing problems.
  • It looks a lot like Swift, but without the hard parts. Call it a 50% of a 90% Swift, without the other 90%.
  • The parser is built to be extended at runtime. Building new features into a language by forcing them into existing syntax seems to just make a mess of your source code. If I want to have something like async/await, I'll extend it into the language instead of making a dance of calls and wrapper types. Source code should be clean.
  • I'm skipping a lot of the complications for now. There is no FFI (I don't need one) and no separate compilation (I don't need to). The parser is slow, it is a combinator based recursive descent which is supposed to be a Packrat Parser, but until I notice it taking up time the rats can wait. I build an AST which directly matches the parse, then walk it and create a whole new AST to generate the code (with a slightly different shape). Coming from single pass compilers with no AST back during the day, this seems colossally wasteful, but after buying big enough hardware to build Swift over and over during my porting attempt, I have cpu cycles.
  • Using the parser is gorgeous! Swift operators and Unicode characters in the source file make it a bit like reading BNF. After each rule there is a binding from its products to variables which are used to build an AST node as a product. Very clean.

Status?

  • integers, characters, strings, bools, arrays
  • structs, classes, enums, functions, methods, closures
  • all the usual flow of control, including defer (there went a week of re-engineering)
  • next up? reference counting (class instances are currently immortal).
  • and then start writing some real programs instead of test programs and see what's needed.

On Writing an Operating System

While pondering what progress could be made in programming languages, I delved into how one might write single process, concurrent code to build systems with many simultaneous clients. It seemed that a stumbling block to feeding such a system was the weird mismatch with Unix and its mostly blocking API.

So, I did the only reasonable thing and started to build my own operating system. It started simply enough with a visit to OSDev.org and wondering if I could just make something say Hello World., and then maybe some keyboard input… and a VT console emulator (multiple virtual), well then I needed serial ports to send out diagnostics, and then a TCP/IP over Ethernet. All that meant I needed a filesystem like thing to store my executables and loaders for the executables and some interpreters to run… Lua, then Wren…

Mini-Moral: Don't visit OSDev.org unless you have a few free years.

Stubborn Guiding Principles of My OS

  • Force concurrent programming to be the best way. This is to force me to encounter the warts and problems and resolve them or document them and live with them.
  • Don't have a full C Runtime Library and POSIX. This will keep me from falling back into the easy, decades old solutions.
  • I can make C do concurrent programming, but it really hurts me. For the low level code user mode code I wrote a concurrent programming library for C, but it is really a pain to use it. This is not the answer, but it let me try out some program architectures by implementing the various OS bits and bobs in different styles, if somewhat clumsily.
  • I want the user mode programs to be in a safe, high level language. There just isn't much point in programming in a language that finds bugs at runtime when you build cycle is "build new OS image, reboot computer, test". There isn't much point in a language that finds its bugs at run time anyway, but certainly not here.
  • Communication between user mode programs must be fast and clean. Ideally there will be zero OS intervention in the communication. (except some scheduling when a process blocks on all of its IO options). I don't want IPC costs to impact the architecture. If lots of heterogeneous processes is the answer, then that's the way I'll go.

Questions You Might Have

What is it called?

For a long time it was named os. But I recognized that was going to be a ungoogleable. Eventually I spent a day and named it "osy", because that was short, googleable, a nod to various simple OS names over the years, and clearly one better than OS X. Apple changed their OS name and ruined my joke. I'm keeping the name.

What kind of OS is it?

I don't want to call it a micro kernel, but the kernel does processes, memory management, scheduling, some primitive bootstrap facilities that are done better in user mode once we get up, and a couple of devices that kind of have to be done in the kernel (interrupt controllers and timers).

Have you ever heard of L4?

Yes. I wasn't aiming that way, but I did land pretty close to L4.

What is the kernel written in?

The kernel is ~6600 lines of C and 400 lines of assembler. That for ia32 and x86_64 architectures, though I'll throw out the ia32 next time I'm in there.

I use a number of Clang C extensions to great effect. Between cleanup and overloadable I end up with much more readable code, and cleanup in particular eliminates a whole class of errors.

You seem to have left out a lot of stuff, like devices.

Not a question, but yes. That is all done out in user mode processes. Not untrusted, if you are going to give a process enough access to enumerate your PCI address space, then you are trusting it.

The goal is to make use of the large numbers of cores available in modern systems by splitting tasks among them vertically as well as horizontally. IPC is cheap by design, and we'll see where that leads us in the solution space.

For now Ethernet, IP, ARP, UDP, and TCP are each a separate process communicating as needed. Surely that isn't the ultimate answer, but it lets me stress the communication. Oddly, it might be the answer for the general case, but acknowledge that UDP and TCP arrive as if by magic through off chip accelerators on high performance network cards.

There still seems to be a lot missing.

Most of the work ended up being in user mode. There I struggle with different models of concurrent programming and try to decide what I want.

I've been through C, Lua, and Wren for my user mode code. Device drivers are all in C, but are pretty simple in architecture so that works out. Lua was quick to port, but ultimately it isn't a sturdy enough language to make me happy. Wren was trivial to port and I like it, but I keep having odd performance issues I can't pin down and I've decided I need a real compiled language if I'm to make any judgements about performance.

I spent a couple weeks trying to port Swift to the OS, but was ultimately defeated by the thousands and thousands of lines of CMake with endless build variants and strange special case rules for OSes. So I went back to the "language" that started this blog, started over using Swift to write the compiler, and will use that to make my user mode programs. I just need to understand what the language is generating, and be able to alter the language to handle the concurrent communication without descending into a see of language warts.

more articles