Blog problems due to Upgrading to Mac OS X El Capitan (10.11 beta)

My blog has been down for the past 24 hours because I decided to upgrade my system to the Mac OS X El Capitan beta release. I first upgraded my Macbook Pro laptop which went smoothly. Unfortunately that wasn’t a good indicator for how the upgrade of my Mac Pro server would proceed.

The first problem was due to an old driver for my CalDigit FASTA-6GU3 (not Pro) PCIe card. After the first attempt to upgrade my server it consistently crashed while booting when it tried to load the USB driver for that card. After doing the upgrade a second time from a USB stick my server would boot; albeit reporting that it would not load the old unsigned kernel extension (kext) for that PCIe card.

Then I had to integrate my old Apache HTTP server config into the new default config file. At that point the Apache web server would again start and serve my static pages. Unfortunately it wouldn’t serve any WordPress page that required access to the MySQL database. Due to several red-herrings it took me a long time to realize that I was simply missing a symbolic link:

sudo mkdir /var/mysql
sudo ln -s /tmp/mysql.sock /var/mysql/mysql.sock

During the process of figuring out why WordPress couldn’t access my MySQL database I decided to update all my HomeBrew installed software (which included MySQL server). That meant several more hours debugging problems with HomeBrew. Starting with the need to update the Xcode developer tools to a beta version. Something it would have been nice if the El Capitan installer had warned me about.

Configuring EasyDNS dynamic DNS updating on asuswrt-merlin router firmware

As part of switching from AT&T to Comcast I decided to upgrade my router. Since I was already using an older Asus router running Tomato firmware I opted for the Asus RT-AC68U. And rather than flashing Tomato on it I decided to try asuswrt-merlin which is basically the stock Asus firmware with bug fixes and some useful enhancements.

One of the useful enhancements is support for custom scripts to deal with situations the GUI doesn’t handle. One such situation is if your dynamic DNS service isn’t supported by the firmware. In my case that is EasyDNS.

Since the wiki article on writing a custom DDNS script didn’t include an example for EasyDNS I decided to document my solution in this article. Per the asuswrt-merlin documentation the following script needs to be placed in /jffs/scripts/ddns-start:

#!/bin/sh
#
# This script provides dynamic DNS update support for the EasyDNS service on
# the Merlin asuswrt router firmware.
#
# This script takes one argument: the WAN IPv4 address.
# It should be installed as /jffs/scripts/ddns-start.
#
# See https://github.com/RMerl/asuswrt-merlin/wiki/Custom-DDNS for more
# information about the context for this script.
date >> /tmp/ddns-start.log
echo "$#: $*" >> /tmp/ddns-start.log

# This should be the domain (or hostname) to be updated.
DOMAIN='Your-Domain-Name-Here'

# This should be your EasyDNS user name and the update token obtained from
# EasyDNS.
EASYDNS_USERNAME='Your-Username-Here'
EASYDNS_PASSWORD='Your-EasyDNS-DDNS-Update-Token-Here'

# Set wildcard "on" if you want this to map any host under your domain
# to the new IP address otherwise "off".
WILDCARD=off

URI_BASE='https://members.easydns.com/dyn/dyndns.php'
WAN_IP=$1

curl --silent --user "$EASYDNS_USERNAME:$EASYDNS_PASSWORD" \
    "$URI_BASE?wildcard=$WILDCARD;hostname=$DOMAIN;myip=$WAN_IP" \
    > /dev/null 2>&1

if [ $? -eq 0 ]; then
        /sbin/ddns_custom_updated 1
else
        /sbin/ddns_custom_updated 0
fi

The next challenge is figuring out how to setup and maintain a AAAA DNS record that maps the IPv6 address for my Mac Pro server to a public host name.

WordPress spammers finally get smart and simulate a real human

Today I saw the first attempt by malware to write a spam comment on my WordPress blog that was not a blatant attack. Today’s attempt was a perfect emulation of a human registering an account at my blog in order to post a comment (note that I don’t require registration to post a comment). That emulation included appropriate delays between the relevant HTTP requests and spanned four minutes and eleven seconds. In fact, the attack looks like it was conducted by a real human whose activity was proxied by a malware infected server.

The attack came from 198.15.148.67 which is owned by serveryou.com, a virtual private server provider (i.e., hosting service). The comment was quarantined by Akismet as spam. When I reviewed the comment it was obviously an attempt to drive traffic to another site via SEO manipulation and was completely unrelated to the article the comment was attached to.

So far I’ve seen only two things in the attack that might be used to detect this type of attack:

1) The requests specified this HTTP header:

Accept-Language: zh-CN,zh;q=0.8

My blog is entirely in english. This header strongly suggests the native language of the author of the malware is a Chinese dialect.

2) The request sequence included several GET requests of /wp-admin/* paths culminating in a POST /wp-admin/admin-ajax.php. The POST data was

interval=60&_nonce=4ba711a06f&action=heartbeat&screen_id=profile&has_focus=true

Prior to this attack my logs show that the action=heartbeat POST to admin-ajax.php has never been seen other than by legitimate requests from myself while working on my blog.

This is the rule I’ve added to flag this attack:

# Beginning 2015-07-26 sophisticated malware that spams WordPress
# comments by doing a very good impersonation of a real user hit my
# site. One obvious mistake the spammer made was in specifying only
# Chinese as an acceptable language.
#
# Note that the Baidu crawler specifies "zh-cn" for this header so
# it's important that the log monitoring program include heuristics to
# avoid blocking acceptable crawlers just because they trip this rule.
#
# Obviously this is a trivially bypassed rule but, again, it's meant
# to quickly reject requests from malware written by idiots.
RewriteCond %{REQUEST_URI} =/wp-login.php [NC,OR]
RewriteCond %{REQUEST_URI} =/wp-admin/admin-ajax.php [NC]
RewriteCond %{HTTP:Accept-Language} !=""
RewriteCond %{HTTP:Accept-Language} !en
RewriteRule ^ /blocked.php [END,E=error-notes:no-supported-language]

My Apache log monitoring program will then blacklist the source when it sees the 400 status returned by /blocked.php. I’ll probably also add a heuristic to my log monitoring program that detects admin-ajax.php and wp-login.php POST requests from “cloud” sources. This, obviously, will only be as good as my ability to identify cloud, virtual private server, hosting providers. Ideas from readers of this article are welcome.

Goodbye AT&T + Sonic.net DSL, hello Comcast Xfinity cable broadband

Short version:

I switched from one evil monopoly (AT&T) to another (Comcast). My Internet connection speeds increased by a factor of 75 (1.2 Mbps to 89 Mbps) and the cost dropped 28% ($70 to $50).

2015-07-25: A Sonic.net representative reached out to me to ask if I had contacted them about upgrading my service. So I’ll clarify that I was already paying for the highest speed tier they offer. Which until recently provided me the approximately 5.5 Mbps download speed I was paying for. A speed fast enough for streaming video from services like Netflix. I am reasonably certain the problem was not with the Sonic.net service but rather with the AT&T infrastructure between my residence and Sonic.net (see below for why).

Long Version:

When I moved to Santa Clara, CA seven years ago I didn’t want cable or satellite television. I only wanted high-speed Internet access. So I chose a setup similar to what I had in my previous residence: DSL (digital subscriber line) service from the local telco (AT&T) and Internet connectivity from a local ISP (Sonic.net).

Until roughly six months ago I was satisfied. The download and upload speeds weren’t as high as those advertised by Comcast but were adequate and I valued the competency of Sonic.net. Also, even though AT&T repeatedly tried to cram unwarranted charges onto my bill (e.g., for long distance phone calls) I chose to stay with AT&T + Sonic.net DSL rather than switch to the most hated company in America.

Then my download speeds started to degrade and were highly variable. At first it was just an occasional pause while streaming a video from Netflix or Hulu. Then it became irritating errors and pauses accessing my Google mail. And finally it became obnoxious long pauses every few minutes while trying to stream a video from HBO or Netflix. I didn’t call Sonic.net because I was reasonably certain the problem was with the physical connection from my home to AT&T. Somewhere between my home and Sonic.net a transmission line or connection was deteriorating. I didn’t call AT&T because my past experiences dealing with them on such issues was so frustrating I’d rather get a root canal. And, most importantly, Sonic.net was no longer adding DSL customers. They were focusing on expanding their fiber optic network. Unfortunately that did not appear to include Santa Clara, CA in my lifetime. It was clear to me that existing DSL customers like myself were on life-support.

So I decided to ditch AT&T + Sonic.net. Yesterday the Comcast installer dropped a line to my home. After reconfiguring my router I find that my download speed is approximately 75 times faster and my upload speed is 10 times faster for a monthly cost that is 28% cheaper.

After the change to Comcast I spent several hours streaming videos from Youtube, Netflix, Hulu and HBO and didn’t notice any pauses in playback. Similarly my gmail access has not had any unexpected pauses or error messages. I was willing to live with an occasional pause streaming high-definition video. I was not willing to live with long pauses while doing something as simple as switching from one gmail folder (i.e., tag) to another.

Will Comcast convince me its reputation as the most hated company in America is justified or surprise me with (relatively) trouble-free service for the next year? Time will tell.

2015-07-25: A Sonic.net reached out to me to affirm that they do not hijack customer’s internet connections to deliver web content. So I’ve removed my assertion that Sonic.net had “descended into evil company territory”. Which leaves AT&T as the likely culprit of that obnoxious behavior. I’m reasonably confident that this is not the result of a malware infection because a) I’m browsing using Chrome on Mac OS X and b) I use [Little Snitch](https://www.obdev.at/products/littlesnitch/index.html) to monitor all of my traffic.

Malware now guessing WordPress credentials via “POST /xmlrpc.php”

Two days ago I started seeing a massive increase in attacks trying to guess WordPress credentials via the xmlrpc.php module. Between 2015-01-01 and 2015-07-10 I saw 155 attempts to guess WordPress credentials via “POST /xmlrpc.php” requests. And 114 of those attempts occurred between 02-05 and 02-22. Those 155 attempts represent an average of 0.8 per day. Since 0300 hours on 07-11 I’ve seen 215 attacks of that nature. For an average of 87 per day — a two order of magnitude increase.

The attacks resemble this one (the most recent from my logs):

POST /xmlrpc.php HTTP/1.1
Connection: Close
Content-Length: 219
Host: www.skepticism.us

<?xml version="1.0" encoding="iso-8859-1"?>
<methodCall>
  <methodName>wp.getUsersBlogs</methodName>
  <params>
   <param><value>krader</value></param>
   <param><value>freefree</value></param>
  </params>
</methodCall>

I don’t know of any way to block these apriori; i.e., before the RPC is actually executed. What I do instead is log the authentication failure. I’ve modified wp-includes/class-wp-xmlrpc-server.php to include the apache-setenv call in the diff shown below:

@@ -237,6 +237,7 @@ class wp_xmlrpc_server extends IXR_Server {
                if (is_wp_error($user)) {
                        $this->error = new IXR_Error( 403, __( 'Incorrect username or password.' ) )

+                       apache_setenv('error-notes', 'auth-failure');
                        /**
                         * Filter the XML-RPC user login error message.
                         *

I then log the error-notes value by including %{error-notes}e in a custom LogFormat. My log monitor then notes the “auth-failure” and blacklists the source. That way the infected computer is unable to launch additional attacks. Obviously in a highly distributed attack of this nature that is of limited value but it’s better than nothing.

Hackers have been abusing ToR since June 27 at an alarming level

A week ago I noticed a significant number of attacks against my web site originating from ToR (The Onion Router) exit nodes. I looked more closely at my logs and found that starting on June 27 every single request from a ToR exit node has been an attempt to guess WordPress account credentials. Those attacks represent 57% of the requests I’ve recorded since the start of 2015 and occurred in just the most recent 6% of that interval.

It is clear that one or more hackers have recently decided to exploit the ToR network to hide their tracks. Which is a shame since ToR is extremely useful to real people trying to avoid nation states spying on them. Nonetheless, I’m still going to block for thirty days any non-whitelisted source that attacks me. Since I don’t plan on whitelisting ToR exit nodes that means people with a legitimate reason to use ToR might not be able to access my blog. Which makes me sad but the security of my web site is more important.

If I ever come face to face with a hacker I would find it hard to keep in check my impulse to beat the shit out of them.

How to Determine If Your Religious Liberty Is Being Threatened in Just 10 Quick Questions

I can’t take credit for this post. It came to my attention via a comment at the blog Why Evolution Is True. That lead me to the Daily Kos article which was the source of the WEIT comment. That in turn lead me to the original article by Rev. Emily C. Heath. It’s a shame more religious people aren’t as reasonable and rational as Rev. Heath.

It seems like this election season “religious liberty” is a hot topic. Rumors of its demise are all around, as are politicians who want to make sure that you know they will never do anything to intrude upon it.

I’m a religious person with a lifelong passion for civil rights, so this is of great interest to me. So much so, that I believe we all need to determine whether our religious liberties are indeed at risk. So, as a public service, I’ve come up with this little quiz. I call it “How to Determine if Your Religious Liberty Is Being Threatened in Just 10 Quick Questions.” Just pick “A” or “B” for each question.

1. My religious liberty is at risk because:

A) I am not allowed to go to a religious service of my own choosing.
B) Others are allowed to go to religious services of their own choosing.

2. My religious liberty is at risk because:

A) I am not allowed to marry the person I love legally, even though my religious community blesses my marriage.
B) Some states refuse to enforce my own particular religious beliefs on marriage on those two guys in line down at the courthouse.

3. My religious liberty is at risk because:

A) I am being forced to use birth control.
B) I am unable to force others to not use birth control.

4. My religious liberty is at risk because:

A) I am not allowed to pray privately.
B) I am not allowed to force others to pray the prayers of my faith publicly.

5. My religious liberty is at risk because:

A) Being a member of my faith means that I can be bullied without legal recourse.
B) I am no longer allowed to use my faith to bully gay kids with impunity.

6. My religious liberty is at risk because:

A) I am not allowed to purchase, read or possess religious books or material.
B) Others are allowed to have access books, movies and websites that I do not like.

7. My religious liberty is at risk because:

A) My religious group is not allowed equal protection under the establishment clause.
B) My religious group is not allowed to use public funds, buildings and resources as we would like, for whatever purposes we might like.

8. My religious liberty is at risk because:

A) Another religious group has been declared the official faith of my country.
B) My own religious group is not given status as the official faith of my country.

9. My religious liberty is at risk because:

A) My religious community is not allowed to build a house of worship in my community.
B) A religious community I do not like wants to build a house of worship in my community.

10. My religious liberty is at risk because:

A) I am not allowed to teach my children the creation stories of our faith at home.
B) Public school science classes are teaching science.

Scoring key:

If you answered “A” to any question, then perhaps your religious liberty is indeed at stake. You and your faith group have every right to now advocate for equal protection under the law. But just remember this one little, constitutional, concept: this means you can fight for your equality — not your superiority.

If you answered “B” to any question, then not only is your religious liberty not at stake, but there is a strong chance that you are oppressing the religious liberties of others. This is the point where I would invite you to refer back to the tenets of your faith, especially the ones about your neighbors.

In closing … remember this: Religious liberty is never secured by a campaign of religious superiority. The only way to ensure your own religious liberty remains strong is by advocating for the religious liberty of all, including those with whom you may passionately disagree. Because they deserve the same rights as you. Nothing more. Nothing less.

New malware using curl to guess WordPress credentials

It appears we have a new script kiddie on the loose.

On 6/16 I saw 35 “POST /wp-login.php” attacks to guess WordPress account credentials with a user agent of “curl/…”. I saw another 37 attack on 6/17. I saw no such attacks going back to the start of the year (when I started tracking such things). After the attacks on those two days I didn’t see another one till today. In the past hour I’ve seen 14 such attacks. The only requests from the attacking addresses are the POST requests.

All of the attacks have been from servers owned by cloud, VPS, or web hosting providers. So I’m confident these requests really do represent the use of the curl command and are not artifacts of malware randomly selecting a user agent. Also, the requests are missing the expected “Referer” and “Cookie” headers that would be present if the requests were legitimate which is why they came to my attention. Here is the Apache rewrite rules I use to flag these types of attacks:

# Block bogus login post attempts. Most bots are stupid. They don't use a
# modern HTTP header, don't include the referer header, or don't include the
# WP cookie that would be included in a legitimate request.
#
# This is primarily so that the logs show a 400 rather than a 200 status
# (which is reported even for invalid credentionals). Thus allowing us to
# cheaply reject the request. It also makes it easier for our log monitoring
# program to notice the bogus attempt and blackhole the source.
RewriteCond %{REQUEST_METHOD} POST [NC]
RewriteCond %{REQUEST_URI} =/wp-login.php [NC]
RewriteCond %{HTTP_REFERER} !^https?:// [NC,OR]
RewriteCond %{HTTP_COOKIE} !wordpress_test_cookie=WP\+Cookie\+check [NC,OR]
RewriteCond %{THE_REQUEST} \sHTTP/(?:0\.9|1\.0)$ [NC]
RewriteRule ^ /blocked.php [END,E=error-notes:invalid-wp-login-post]

P.S., Yes, the number of attacks is quite tiny. But that’s not surprising given that my blog sees on average approximately 50 legitimate requests each day.