Saturday, June 24, 2006

The world's largest FOSS IRC network, FreeNode, has been (for lack of a better word) hijacked. The culprit, who went by the nickname ratbert, seems to have nabbed the privleges of Robert Levin, President and Executive Director of Peer-Directed Projects Center (FreeNode's parent organization), AKA lilo. Whew! As if that wasn't enough in itself, ratbert pushed out an offensive global message and attempted a DCC SEND exploit. He then proceded to kill and/or k-line every staff member in sight, including lilo, and brought down quite a few of FreeNode's servers. This log shows the ominous beginning of the mess:
-ratbert- [Global notice] I am a fat asshole, who loves abuse, die
The rest is too broad and too long to log in full, but mainly consists of FreeNode staff members being killed (with some colorfully interesting reason messages) and cries of "MAYDAY! MAYDAY!" and other expressions of terror throughout the many channels of FreeNode.

Everyone seemed to be making the most of the situation. Humorous allusions to Star Trek and other movies were thrown into the chaotic pandemonium that was the chat in many of the larger channels that I happened to be in (such as #freenode-social and #gentoo-amd64). The situation really did make one feel rather excited and giddy; it was hard to take the situation seriously, with no real danger to ourselves, a subtle underlying self-assurance that FreeNode was indeed, in the long run, invincible, and no responsibilities of our own to the network (the FreeNode staff, on the other hand, may tell you otherwise). The servers going down (which looked just like a lot of netsplits to those whose servers were spared) seemed to add a bit more terror to the thrilling mix, but in hindsight it didn't amount to much.

The much more stoid moment that will be used to summarize the gravity of the matter came when our beloved lilo was taken down:
* lilo has quit (Killed by ratbert (die ))
Thankfully, our self-assurances have indeed proven themselves: the situation seems to be clearing up as I write. Most, if not all, of the servers have been recovered and relinked to the network, and yes, lilo made his way back. His words do little to clear up the situation further, but are better than nothing:
-lilo- [Global Notice] Hi all. As you may be aware, freenode has experienced a crack attack and we're working on tracking down the details. At this point, we cannot guarantee that more problems will not occur.
This raises some major questions about security. If this guy got lilo's password, could he have ours? This is not just pure speculation:
<dbo> lilo, is there any chance he was capturing passwords from people that were reconnecting or was the attack not aimed at that? there have been 3 or 4 people who said the registered with nickserv even when it was down
<lilo> DBO: it is possible someone turned off the jupes
Woah! If someone did manage to gather people's NickServ passwords, it could mean major trouble, for the victims themselves and possibly for FreeNode as well. Still, this is for the most part speculation at the moment. (EDIT: This is no longer the case; see the UPDATE.) ratbert comes across more to me as some clown that wanted his hour and a half of infamy more than anything else. If he really wanted to steal passwords, he most likely would have done away with the servers quickly and as soon as he got his privleges, rather than spend all that glory time k-lining staff members and risk getting k-lined himself by one that was paying attention. Nothing can be said for sure, however.

Well, looks like we're just going to have to wait this one out. Hopefully we'll get more details in the near future. I'll keep you posted...

The freenode staffers just finished a Q&A session about the matter. Special thanks to Keith Gable (AKA Ziggy on freenode) for hosting the log in full at his site here. Summary of points:
  • Yes, passwords were likely compromised, but they estimate that only 25 or less were actually stolen. Change your password anyway. In the words of a staffer:
<hedgemage> WhiteNoise: there was a small window between the time that nickserv went down and our servers stopped accepting connections. While <25 is only an estimate, we are fairly confident that it is accurate. That said, it is quite easy to change your password so you *know* you are safe.
  • They can't give any specifics on security, how the attacker was able to compromise the network, or suspects.
<hedgemage> We are not releasing our suspect list, but we have some reasons to expect that bantown or GNAA may have been involved.
  • Freenode is still analyzing the matter and will release the above information when they can. They don't want to compromise the network or any possible future law enforcement investigations.
<astinus> We can't comment on matters of security, anything said might taint investigations by any law enforcement authorities in the near future. We are looking into this, we are serious about finding the root cause of this, and we have your security in mind
  • They're still looking into the matter of whether or not a crime was actually committed. I got the feeling that they were looking to press charges, but we'll see.
<HedgeMage> JapaneseGangster: While we can't, right now, comment on security measures that aren't in place yet, we need to assess our vulnerability and whether a crime was committed. We don't, at this time, have evidence of enough damage for that to be the case.
  • (EDIT: Added) lilo apologizes for his request for donations right after the servers went up, if you wanted one. I personally think he was trying to add humor to the situation, but obviously it didn't come across too well for some people.
<HedgeMage> Re: the notice regarding donations, lilo has asked me to apologize if anyone was offended

This is just about all we could get out of them. Unfortunately, the staff decided to cut the session short due to several trolls (including, it seems, our friends at GNAA), and questions that kept being repeated. Sorry guys, but it looks like they weren't going to release much more information anyway. I'll still be keeping you posted as this develops further. Until next time...

Note: Before you comment, pease refrain from continuing the groundless attacks on lilo and the unnecessary vulgar and offensive language. We don't need it and we don't want it here. Any posts violating this may be rejected by me. Don't "flame" people without the necessary factual support, and keep your language clean. I don't want to be a big bad control-freak moderator. Unfortunately, the amount of comments that are on the level that I have described is incredible. The blog has obviously been targeted. I've been forced to have all comments pass through me before being posted; it should prevent you from having to register. Basically, if your comment doesn't add to the story, don't post it. Thank you.


Anonymous Anonymous said...

I'd have renamed him grub before dropping the k-line.

4:31 AM  
Anonymous One of freenode's channel operators. said...

Well it made slashdot.

11:30 AM  
Anonymous Aaron said...

Both of these annonymous comments show their mentality and maturity level. Good job guys.

My question is this: what security threat does this really pose? If your nickserv password is stolen, then you just register a new nick. If you use the same password on freenode that you use for other services, like email and blogging, then you better change your password.

The actions of this guy don't seem to be permament, although he may have done some temporary damage at the time. Big news though, no doubt.

11:32 AM  
Anonymous Anonymous said...

The problem is that people know you by your nick so getting a new one is fine if no one knew you. I have ops in quite a few channels and I'm going to be watching my account closely and as soon as the staff confirm things have been cleaned up I will be changing my password. I'm still not too found of the sending of them to nickserv in plaintext anyway. :(

The above mentioned op.

11:49 AM  
Anonymous Anonymous said...

Many of these online communities, such as YTMND and Freenode, ask for "donations". Where is this money going? Is it going into the pocket of the guy who runs everything? I demand accountability from online sites that demand donations. Is this money buying lilo free pizzas every night and vacations to europe for Max Goldberg of YTMND?

12:00 PM  
Anonymous Anonymous said...

Yeah, some accounting from FreeNode would be very interesting! I've ran quie a few IRC servers over the years, and the amount of funding they get covers plenty more than just hte traffic!

12:17 PM  
Anonymous Anonymous said...

well from what i read from that rapidshare link, lilo gets paid twice. thats strange, anyone care to comment?

1:15 PM  
Anonymous Anonymous said...

No need to /msg nickserv btw, you can also use /quote nickserv which uses the ircd's NICKSERV command feature, which'll tell you if services are down instead of sending the message to a user using the nickserv name.

On a network I help running (with a modified bahamut) opers are only allowed under SSL, which may help against network sniffing (but definitely can't do anything against social engineering or keyboard codes logging, obviously).

As for using an HMAC (i.e. HMAC-SHA-160) or other such system to register with services to avoid having passwords sent in plain text over the network, this would unfortunately require a client modification. However, this would be trivial to implement if everyone agreed to modify their client to support the new command and IRCd software patched.

1:31 PM  
Anonymous Anonymous said...

and you wonder why #debian left freenode.

2:47 PM  
Anonymous Anil said...

Lilo has the "give me a freeride" gene:
[1151212158] -(01:09:18)- [freenode] -lilo(i=levin@freenode/staff/pdpc.levin)- [Global Notice] Once again I want to take the opportunity to mention that
freenode could use more of its own hardware resources and more server hosting resources. If you'd like to help, please email ....

As if the $25,000 that he made last year wasn't enough...he is just wants money ;P

5:35 PM  
Anonymous Anonymous said...

Seems to be alot of drama for an IRC server.

7:20 PM  
Blogger HeavensBlade23 said...

Story made digg.

7:56 PM  
Anonymous Anonymous said...

wow $25,000/yr is nice for a man on disability with his wife who both live in a trailer in houston. lilo has a sweet life.

8:02 PM  
Anonymous Daniel Robbins said...

Look at the type of jerks that Rob Levin is forced to deal with on a daily basis. It's a miracle that he continues to put in the time and effort. Thank you, Rob.

8:32 PM  
Anonymous Anonymous said...

wow talk about having a blog that got super successful in like 10 minutes

9:02 PM  
Blogger ptr panda said...

love the emphasis on terror. so i guess this guy is a TERRAHRIST! wont anyone think of the IRC CHILDREN?!?!?!

10:30 PM  
Anonymous Anonymous said...

My take on this?

lilo's a noob. He's had passwords social-engineered and now cracked out of his network enough times that he should probably read up on security (and verify user identities before he hands out O:lines).

On the other hand, $25000+disability for a couple isn't much (why do you think they live in a trailer?), so I don't know what people are bitching about lilo getting paid.

Generally I'm sure lilo's an okay guy, but he should have learned a bit more before running one of the world's largest IRC networks.

Also...people shouldn't take this so seriously. It's IRC. If you make your entire living (or even most of it) from IRC (with IRC being a crucial part of your living) you probably need to find something better to do.

I personally find the whole thing pretty funny. This is why I love EFnet - stupid stuff can happen all the time and nobody cares, nothing gets posted to Digg or Slashdot, nobody gets too up-in-arms, and it's good clean e-fun all around.

11:02 PM  
Blogger Andrew said...

Well, I'm an chanop on all of the Wikipedia channels, you can damn well bet I'm changing passwords, so whatever. Rob was really good about it, I had him on privmsg for pa fair bit of it and he was great. Just amazingly great

11:17 PM  
Blogger MakoMK said...

Hmmm... before all hell broke loose, something weird happened:

Jun 25 03:32:57 * lilo has quit (Killed by Astinus (Please do not use tor as an abuse medium for GNAA activities.))
Jun 25 03:33:39 * lilo (i=levin@freenode/staff/pdpc.levin) has joined #mythtv-users
Jun 25 03:33:49 Dagmar Just a reminder: No one ever said getting MythTV running wouldn't entail learning a lot about how it's various parts actually work
Jun 25 03:33:56 Dagmar Learning is good for you.
Jun 25 03:35:55 smokey actually, i found this in the backend log... select timeout - ivtv driver has stopped responding
Jun 25 03:36:11 Dagmar wait aminute
Jun 25 03:36:15 Dagmar Did I read that corrrectly?
Jun 25 03:36:33 smokey yeah, i cut and pasted it
Jun 25 03:36:34 * bilbravo ( has joined #mythtv-users
Jun 25 03:36:35 Dagmar ... SignOff lilo: #mythtv-users (Killed by Astinus (Please do not use tor as an abuse medium for GNAA activities.))
Jun 25 03:36:49 Dagmar smokey: I was referring to lilo's quit message.
Jun 25 03:37:26 Dagmar Astinus is in for it now. :)
Jun 25 04:13:33 lilo (whoops, Astinus mistabbed 8)
Jun 25 04:13:44 Dagmar hehe
Jun 25 04:43:49 * lilo has quit (Killed by ratbert (die ))

4:15 AM  
Anonymous Anonymous said...

501c3's are required by the IRS to post their returns, if you want to know where the $25k/yr went last year...STFW and stop complaining. Your obviously fanboy's of the network if you know enough to complain. But just like every other fanboy you'd rather bitch about a non-existant problem (accountability) than do a little research.

8:52 AM  
Anonymous Anonymous said...


in the US is way below poverty and is not alot of money, maybe he should "hire" or get in good with someone who knows security

9:41 AM  
Anonymous Anonymous said...

i am telling you it was not GNAA you dumbass. GNAA does nothing but idle on their stupid irc server, which has been taken over too. ask any current member about the april 1st takeover.

11:42 AM  
Anonymous Anonymous said...

Before the press conference #freenode-moderated was unmoderated and many people were there discussing the attack. One of the points someone made in response to the slashdot story that "ratbert" is not, in fact, an attacker, but is an alternate nickname for lilo. That would imply that lilo's account was indeed hacked.

This is just what I heard, I can't recall by who, but you can independently verify it on freenode if you wish.

2:04 PM  
Blogger MoiraA said...

I don't know why some people are using this as an excuse to have a go at lilo. I'm in one of the same channels on freenode as he is and I've always found him a really nice guy. He was great when I had problems with a rather objectionable user and although lilo must be really busy and in lots of channels, he's got time to talk to everyone.

4:49 PM  
Anonymous toxik said...

I guess those outright morons hacking on lilo got something to think about now - I mean with the accident and all.

If I am to say my opinion then I'd say that exploits and such does happen. Governments have been cracked too you know, and,

Rest in peace, Rob "lilo" Levin

5:32 PM  

Post a Comment

<< Home