I have a problem with people who don’t manage their web content. Many years ago, we built a web site for a non-profit organization. Then, a year or two later, the owner asked permission for a friend, maybe a patron, to rebuild the site with the latest whizz-bang Drupal, and still leave it hosted on our server. We reluctantly agreed and let him do his thing. A few years later, the client requested some changes, but could not reach the Drupal expert. We contacted the expert, but he was too busy to do the work. I did some reading up on it and got into the site and found not only was the site full of spam, but the version of Drupal was two major releases out of date and could not be upgraded, at least not without a wizard. And I don’t qualify.

Well, I saved off all the material I could, and punted Drupal. I rebuilt the site using our old, faithful server-side includes and templates, and ultimately upgraded it to HTML 5 standards. Lesson learned.

Another client installed Mediawiki on his server, and, I guess, pretty much let it go. Some time later, I got into the server to fix a problem or something, and discovered his Mediawiki installation was a spam haven, and several revs out of date. I informed him of the state of things and he requested I clean house and install the latest, and I did. This time, I configured it to require owner moderation of all user input.

As always, comments and corrections are welcome. Email me at the address at the bottom of this page.

FYI, “CAPTCHA” is an acronym for “Completely Automated Public Turing test to tell Computers and Humans Apart”. For more information, see the Wikipedia article.

For my and my clients’ servers, I deal with crackers (cyber criminals the media call “hackers”) and spammers. The way I deal with spammers is covered in another article.

I use fail2ban on all of my servers. It watches the logs and when it sees several authentication failures from one IP, it blocks that IP for a period of time and sends me an email about the action. The notice includes the “whois” information for that IP, and I’m interested in the abuse contact, because I’m going to send a complaint to the contact about abuse coming from his network.

Internet standards, namely RFC 2142 says you should have a working “abuse” point of contact and a “postmaster” point of contact listed in the whois information for your network. When I send a complaint to the abuse POC, I expect them to hunt down and shut down the machine on the offending IP. Some of them send back an auto-reply. Some send a personalized acknowledgment. Some even send a follow-up reporting the results of their findings. One even reported that he “terminated” the rogue machine. I got a kick out of that.

Some don’t send me anything. I don’t mind as long as they deal with the blighter. (No I’m not British, but I really like some Brit-isms.) If my complaint goes to /dev/null, I have no way of knowing. If any IP, or sometimes network, goes on unabated, I block them for a month. I will not have crackers freely hitting my servers.

The vast majority of POC information is available from the text-mode “whois” program, available for every flavor of UNIX and Linux. I’ve found two primary areas that frequently do not provide any POC information. The first is most of the networks registered with AFRINIC, the African Regional Internet Registry (RIR). I don’t know if they don’t provide POC email addresses because they are RFC-ignorant, or are afraid of publishing an address and getting spammed, or are just lazy. My policy is, if they don’t publish an abuse POC and some machine on their network hits my server, the network gets blocked for a month.

A few months ago, I noticed that whois info from Brazilian networks had no abuse POC email addresses. Their stock whois footer lists cert@cert.br, who are also interested in such reports. So I sent a note to them asking what’s going on with that. They forwarded my note to the folks at registro.br which is the Brazilian registry. They pointed me to their web-based whois. I am against web-based tools for this because I build scripts to simplify a lot of my work. If I have to stop, bring up their web site, copy and paste the IP, click several places, and then copy and paste the results to my script, that really bogs things down. But I’m willing to do it on a limited basis.

Well, I tried that, but still didn’t get the email POCs. I informed them and they said if you fill in the CAPTCHA block correctly, then you get the email info. Well I saw that stuff there, but it was accompanied by text in Portugese, so figured it wasn’t important. The CAPTCHA has several images of characters, and a prompt in Portugese. They pointed me to some translation web sites where I could find out what the prompt is. Can you not provide subtitles in English? The whole world does not speak Portugese.

Okay, so I tried it. Didn’t work. Tried several more times and failed. It’s not always obvious if the letters are uppercase or lowercase, and you have to ask if it’s a lowercase “L” or the numeral 1. I complained to them about it and ultimately they said they would pass my complaint to their developers. The final irritation is that it says (in Portugese, of course), “If you have difficulty with the image above, use the version without the image challenge or contact our service.” And, of course, if you do that, you get the restricted whois without any POC emails.

I challenge you to try it. Go to the web site and try a Brazilian IP, say, 186.235.159.62. Answer the CAPTCHA, and then scroll down to see if you get any email addresses for the POCs.

For the nonce, I’m blocking for a month all Brazilian networks harboring botted machines that hit my servers. If I can’t solve their friggin’ CAPTCHA to get their POC emails, then they might as well not have any at all.

Sorry I took so long to get to the point of the title, but I needed to explain how I got there.

But continuing with the subject line, the second worst CAPTCHAs I’ve seen have pictures and you are supposed to click on the ones that do or do not fit some criteria. One had you click on the pictures that had storefronts. Some of them were fuzzy and it wasn’t always obvious what constituted a store front. Another had images of streets and you had to click on the ones with “street signs”. Define “street signs”. I think I went through four of those before I passed.

The most usable CAPTCHA is called “reCAPTCHA”. This is used by the AFRINIC web-based whois. Aside from it being web-based, it is the least objectionable form of CAPTCHA. After you enter the IP into the search box, it puts up a box, and you are to click inside the box. When you do that, it spins for about five seconds and then presents a check mark indicating you are approved. Then you hit “Search” and you get the requested information. As noted before, you still frequently get no POC email addresses, I presume just because they don’t have any in their database.

Update: AFRINIC no longer uses any kind of CAPTCHA. Perhaps they figured if they are (mostly) not going to provide POC emails, then there is no point in using a CAPTCHA to keep email harvesters out.

Let me know if you have differing experiencees.

I’m just trying out Plerd as a blogging vehicle. I reviewed a bunch of possibilities over several years. Obviously it wasn’t a pressing need.

The first place everyone turns for blogging is WordPress. I tried that for a time, but found it to be a maintenance headache. Because it is ubiquitous, it’s a favorite target of crackers[^1], and there seems to be no shortage of holes to exploit, especially in the plugins. I doubt that it is because PHP has so many vulnerabilities. I’d more easily believe PHP coders are just lazy about input validation.

I was going to use Blosxom, and while it’s pretty simple to install, the output is underwhelming. To get all the stuff you want it to do, you need to install plugins, many of them, and in the right order. I wasn’t really keen on climbing that learning curve.

I looked at several others, but many of them lacked features I wanted or hadn’t been touched in 5-10 years. I want something that’s still being maintained.

Plerd made the most sense to me. It’s simple, requiring no special processing on the server, creates a static blog not volnerable to cracks by enterprising criminals, and provides RSS links. Its only shortcoming is that there is no mechanism for comments. Well, that’s maybe a plus, because that’s one less target for crackers. Emailing comments works just fine.

Plerd uses one directory (or Dropbox resource) for the source material and another for the document root. To create your content, you use a common text editor in Markdown language, or use a Markdown editor of which there are many for nearly any platform, include mobile devices. Markdown is kind of a simplified HTML. If Plerd is normally set up to run as a daemon, when you upload something to the source directory or modify a file there, Plerd sees it and converts it to HTML and puts it in the doc root.

If I decide to switch to something else, the conversion won’t be too much of a hassle (he said, confidently :-).

[^1]: The media calls them “hackers”. I consider myself a hacker but not a cyber criminal.

My sysadmin work can probably be boiled down to three or four categories. Of course there is software upgrades and rebooting, as well as researching things like email problems and configuration problems, but the two biggest daily chores are dealing with crackers and spammers. I will deal with the other subjects in other posts, but for this post, I will concentrate on attempted cracks on my and my clients’ servers.

My primary tool for dealing with crackers is fail2ban. It works by monitoring specific logs for authentication failures. If a machine at a given IP tries and fails to authenticate N times in X minutes, it is blocked (usually using iptables) for y minutes, and if so configured, sends me an email about it. The email contains the results of a whois query for that IP. I have a script that I use to scan the related log for that IP and then sends a boilerplate complaint to the applicable network admin.

One of the nicest features in Fail2Ban is the “recidive” jail. Many cracker bots get blocked the first time, and then wait for the f2b block to timeout, and then hit it again. If allowed to continue, they will go on until the world looks level or they guess a username/password. If recidive detection is turned on, the blighter will be put in the recidive jail and be blocked for a week.

The skeptic will say that’s like peeing in the ocean. Sometimes I get back an auto-reply, sometimes a response like, “We have shut down the offending machine”, sometimes an NDR (non-delivery report), sometimes nothing. When I get an NDR indicating that there is no working POC email, I block that whole network for a month if it’s outside the US and Canada. If it’s inside the US or Canada, and there is a telephone contact, I will call them to advise them of the crack attempt and that the email notification failed. If it’s still a working ISP, I usually get an alternate email address to send the information. Sometimes the ISP has changed hands and not changed their contact information, or simply gone under. If I get no satisfaction there, I block them for a month.

A few registries, APNIC in particular, have a link where you can report inaccurate whois information. I use that for NDRs reporting “lost connection” or “recipient unknown”, but never for “mailbox full”. There are several POCs in China whose mailboxes are perennially full. I don’t report them. I just block their networks, again, just for a month.

In the case of NDRs with “recipient unknown”, sometimes I forward the NDR to the postmaster address they are supposed to have with the note:

Please fix your mail server. Internet standards (RFC-2142) say you should have an advertised, working “abuse” mailbox.

But of course, those frequently bounce back with “recipient unknown” NDRs. A working postmaster email is one of the other things strongly recommended by RFC-2142. Well, it’s worth a shot.

Let’s talk about whois contact email addresses. These should always be generic “abuse@whateverdomain.tld” and “postmaster@whateverdomain.tld”, plus other generic terms like “noc” or “admin” and the like. It should never be the specific individual’s address for several reasons. First, the abuse and postmaster addresses should be aliased to two or more real admins in your mail server. That way, if one of them goes on vacation, someone will get the email. Don’t get me started on vacation autoresponders. Second, personnel may come and go. If you have joe.blow@someisp.tld listed in your whois, and he moves on, you now need to update your whois information which is likely much more difficult than modifying the aliases file in your mail server.

I hope these tips are helpful to you either as a consumer of whois information, or as an IT person in your company. As always, send me an email with any questions or comments. And, of course, flames and spam -> /dev/null.

“System Administration: It’s a dirty job but someone said I have to do it.” - on a T-shirt sold by the late, great SysAdmin magazine.

My preferred work is in software engineering, but in every place I developed software, I pretty much had to do my own system administration, usually because they couldn’t justify a full-time sysadmin, or the full-time sysadmin was too busy dealing with Windoze boxes.

In the past, I have administered AIX, HP-UX, SCO Xenix and Unix, Solaris, and FreeBSD. These days it is just Linux. My first Linux was Slackware, and then Red Hat v4.2 — yeah, long before Fedora and CentOS. So, yeah, I think I qualify as a “greybeard”, though I keep it shaved.

My preferred distro is CentOS and nearly all of my and my clients’ machines are running that. The lone exception is my antique laptop which is running two flavors of Ubuntu. I was going to put CentOS on it, but CentOS no longer included drivers for my WiFi card, but Ubuntu does. I have nothing substantive against Ubuntu. It has a little bit different philosophy about some things, e.g. root logins, but mostly administers the same as Red Hat and its derivatives, at least from the command line.

The subject matter for posts to this blog includes some of my practices as a sysadmin, as well as opinions and recommendations. This is a pretty minimalist blogging system so has no mechanism for comments built into it, but you can comment by sending me email (link at the bottom of the page). I will post thoughtful comments at the bottom of the related page. Crude comments and ad hominem attacks will go to /dev/null.

I am a software engineer and system administrator. I’ve been doing software engineering since 1986, first in assembly language, then C, then Perl and an assortment of UNIX tools. I started doing system administration when I got my first UNIX-like OS, SCO Xenix, around 1988, I think. Since then, I’ve worked with SCO UNIX, AIX, SunOS and Solaris, and HP-UX, and finally Linux.

I worked first for a vertical-market software outfit, then got into contracting through several consulting companies. Finally, I incorporated Bobcat Open Systems, and do web development, web design, and web hosting, as well as consulting and contracting.

In this blog, I cover my experiences including recommendations and rants about things going on in my company and the computer industry. I hope you will find them useful.