IPO! Senior leadership and founders rang the bell on the floor of the NYSE...
...and then Dell delivered an ice sculpture...
...and then we took it out on the watermelons of the world...
...and then the founders and senior leadership got back...
...and then it got really hot...
...and then we all went back to work...
...but still made time for WoW.
I can't remember if I mentioned this earlier or not, and quite frankly I'm too lazy to look. So here's the idea. My servers do their internal communication on separate gigabit backbone, leaving the 100Mb NICs free to handle the rest of the traffic. I'd love to have a 10Gb backbone, but I can't afford a Cisco 6500 at the moment. Or possibly ever.
Anyway, the spine takes advantage of Erlang's ability to interconnect nodes running on different machines virtually transparently. The code is nothing more than a complicated wrapper around Erlang's '!' primitive, which is used to send messages between nodes.
My initial use for it is to send relevant entries from the BIND logs on the DNS server to the web server for easy display on the secondary DNS hosting site that's nearing completion. While this is a very simple application, there are plenty of other uses. After a few security-related tweaks, a node will be placed on our back-up cluster in Dallas/Fort Worth for doing backups.
The nice thing about having an intelligent spine is the each node can report back on the status of it's host machine. If, for example, a web server was experiencing difficulties, it could communicate that to a fail-over server which could take over until the problem could be resolved. It's sort of like a cloud concept - there are merely servers that behave however they have to.
Since Erlang nodes can pass instructions to other nodes, the network could essentially re-structure itself if needed.
To get a better idea of what it's like, Kevin Smith has a fabulous series of screencasts on the Pragmattic Programmers site. Go and buy them.
Kevin, if you're reading this and would like to team up on a screencast, let me know.
Last week was the first Rackspace developers conference. While the majority of the people that attended were employed as software developers, a handful weren't. I was one of three people from network security who attended, and as the picture below shows, I apparently was fascinated.
Lessons learned:
Lessons Confirmed
After recently installing Ubuntu 8.04 on one of my boxes, I have only two complaints.
1) The laptop I have apparently has the only unsupported Atheros wireless chipset in history (AR242X).
2) Every two or three minutes, an error shows up in /var/log/auth.log (username removed):
May 13 21:26:03 neon sudo: username_removed : TTY=pts/3 ; PWD=/etc/pam.d ; USER=root ; COMMAND=/usr/bin/vi common-password
May 13 21:26:03 neon sudo: PAM unable to dlopen(/lib/security/pam_smbpass.so)
May 13 21:26:03 neon sudo: PAM [error: /lib/security/pam_smbpass.so: cannot open shared object file: No such file or directory]
May 13 21:26:03 neon sudo: PAM adding faulty module: /lib/security/pam_smbpass.so
May 13 21:26:03 neon sudo: pam_unix(sudo:session): session opened for user root by username_removed(uid=0)
May 13 21:26:03 neon sudo: pam_unix(sudo:session): session closed for user root
This message also appears whenever you sudo anything, filling up the auth log and making it virtually impossible to quickly skim through it and see meaningful messages. The error appears to be related to an auth mechanism that comes pre-enabled for SAMBA. Why that would come pre-enabled is beyond me, but the fix appears to be relatively simple.
In /etc/pam.d/common-password, find the line that says:
password optional pam_smbpass.so nullok use_authtok use_first_pass
...and comment it out. Next, find
auth optional pam_smbpass.so migrate
...in /etc/pam.d/common-auth and comment that out as well.
Done and done. Enjoy your minty-fresh auth log.
I'm studying for my CCNA and while reading a section dealing with how switches learn MAC addresses, a thought occured. The left 24 bits of a MAC is a code unique to each manufacturer while the right 24 bits is a series of bits that manufacturer has never used before (in theory - in large enough runs, you'll get some dups).
I had always assumed that if you connect two NICs with the same MAC to a port security-disabled switch the first one to be learned would be the one to receive all the traffic. My logic was that when the switch was checking it's lookup table to see to which port MAC "X" was connected, it would see the first entry, stop looking, and send it the data - X' I thought would never actually be seen in the lookup table, since presumably the switch would stop looking when it found X.
Which brings me to a tangential subject: I propose that people who use the phrase "It was in the last place I look" should be deported. It's yet another phrase with no actual meaning behind it; most people stop looking when they find something, so naturally it's going to be in the last place you look ("Honey, I found the keys but I'm going to check the couch and the front porch and make sure they're not there!"). Anyway...
The effect, I presumed, would be a 'ghost sender' throwing frames at the switch but never receiving a response. As it turns out, the switch is smarter than that (I didn't REALLY think the people at Cisco hadn't thought of that, it was just an idea).
Each time a frame is received by the switch, it apparently re-learns the MAC. When X sends something via port Fa0/1 the switch says 'Oh okay, you're on Fa0/1 - let me write that down'. Then when X' sends a frame via port Fa0/2 the switch snidely quips 'Wow, you get around, you port-hungry devil, you!' and re-learns the MAC. This is of course, assuming that switches are capable of 'quipping' (the latest research suggests they are - I know they can be snidely).
After that happens, let's assume that the traffic in response to the request X sent earlier shows up. The switch (being worried that the traffic was out that late) says "Where were you, Series Of Bits?! This is not a hotel! Go to your room it's right..." - the switch examines the lookup table - "...THERE!" and gestures angrily at port Fa0/2.
Series Of Bits, obediently goes to port Fa0/2 to find a very surprised X'. X' looks at Series Of Bits with a visage similar to when you haven't ordered a pizza and one shows up anyway. "Anybody order a series of bits??" it hollers. "Yeah, says Thomas J. Program, peaking head through the NIC's door. "But not THAT series of bits."
Meanwhile, poor, poor X is waiting for a Series Of Bits that never comes. We can only assume it begins to cry. The point of all that (other than that I'm CLEARLY single if I had the time to write all that) is that when two identical MACs are connected to the same switch bot experience seemingly abruptly random traffic as the switch learns and re-learns the MAC. Why the hell didn't I just SAY that? Because I was bored. But now I'm not.
Note: I was tempted to give the NICs a Brooklyn accent and make a "New York NICs" joke, but decided against it. You can thank me in US Dollars.
I mentioned in the last entry that Jan Lehnardt was also talking about gen_server, and his article is now up:
Read it here.
Those of you playing along at home may have gotten the impression that I kinda like this whole Erlang thing. Correct. Those of you playing along at home (and wayward surfers as well) from outside of Europe may be wondering why there's a bothersome 'u' lurking in the spelling of 'behaviour'. Erlang came from Ericsson (who denies the connection between 'erlang' and 'ERicsson LANGuage'. They claim it was named after a mathematician of the same name.) and Ericsson just so happens to reside in Scandanavia.
Linguists theorize that they are allowed to put the letter 'u' wherever they please as a trade-off for having to use that weird o-with-a-line-through-it letter. Or maybe they just want to show the word how much they love empty sets. Regardless, that being said let's hope into gen_server.
gen_server is a component of OTP, the Open Telecom Platform. OTP can be thought of as a sort of application framework for Erlang, much as Python has Pylons, Ruby has Rails, and Windows has Problems Crashing. OTP is hilariously powerful, and being the relative new-comer that I am, I've only just started to really grasp just how powerful it is. To quote Joe Armstrong, author of the fantastic book "Programming Erlang":
The power of OTP comes from the fact that properties such as fault tolerance, scalability, dynamic-code upgrade, and so on, can be provided by the behavior itself. In other words, the writer of the callback does not have to worry about [those things] because this is provided by the behavior.
Okay Philip, what's a behaviour? A behaviour is just what it sounds like - a bucket containing all the common behaviors of a certain kind of system or platform.
In the great tradition of contrived examples, imagine a movie store that would only let you check out one movie at a time. The code for that might look something like what I've included below. Just like last time, first the code and then the explanation. My eternal thanks go out to Jan Lehnardt of the CouchDB project for his assistance, patience, and friendship. In the near future, he's going to be posting a plok entry on this same subject, so by all means go see if that's there. And give him money.
-module(movie_store).
-behaviour(gen_server).
-export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2, code_change/3]).
-export([checkout/2, lookup/1, start_link/0]).
start_link() -> gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
checkout(Customer, Movie) -> gen_server:call(?MODULE, {checkout, Customer, Movie}).
lookup(Customer) -> gen_server:call(?MODULE, {lookup, Customer}).
init([]) ->
Tab = ets:new(?MODULE, []),
{ok, Tab}.
handle_call({checkout, Customer, Movie}, _From, Tab) ->
Response = case ets:lookup(Tab, Customer) of
[] ->
ets:insert(Tab, {Customer, Movie}),
{ok, Movie};
[{Customer, OtherMovie}] ->
{already_checked_out, OtherMovie}
end,
{reply, Response, Tab};
handle_call({lookup, Customer}, _From, Tab) ->
Reply = case ets:lookup(Tab, Customer) of
[{Customer, Movie}] ->
Movie;
[] ->
none
end,
{reply, Reply, Tab}.
handle_cast(_Msg, State) -> {noreply, State}.
handle_info(_Msg, State) -> {noreply, State}.
terminate(_Reason, _State) -> ok.
code_change(_OldVersion, State, _Extra) -> {ok, State}.
Okay - it looks terrifying (which incidentally, is Erlang's super power) but it's not. The first line declares the module name (and 'movie_store.erl' should also be the filename) and is a standard part of every Erlang program.
-behaviour(gen_server).can be thought of a line that declares which 'template' we'll be using for this program. The gen_server behaviour mandates that we define the functions specified in the first set of exports.
-export([checkout/2, lookup/1,start_link/0]). makes the custom functions that the server uses available to the outside world. They are in a separate -export statement only for clarity and can be combined with the one above.
start_link() -> gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
The starting point for the program. This function as you can see, serves as a wrapper for the command that spins up the server. {local, ?MODULE} shows that the server will be run locally and will not be available to other Erlang nodes in the cluster (as opposed to global). ?MODULE is a stand-in for the name for the current program name and specifies what we'll use to reference the server. The second ?MODULE tells the server where to find the callbacks (in this case, the same file). The remaining options allow you to turn debugging on, log to a file, and a few other things not used in this example.
checkout(Customer, Movie) -> gen_server:call(?MODULE, {checkout, Customer, Movie}).
lookup(Customer) -> gen_server:call(?MODULE, {lookup, Customer}).
These lines tie in the custom functions and tell gen_server what to do when (in this case) movie_store:checkout(...) and movie_store:lookup(....) are called. ?MODULE indicates where the function is, and {some_name,Arg1, Arg2,...} gives the function name and the arguments to be passed.
init([]) ->
Tab = ets:new(?MODULE, []),
{ok, Tab}.
creates an database table using ets, the built-in memory database and returns it. In a deployed program, this would probably be something a bit more...permanent. A key component of a gen_server program is the program state. In this example, you'll see Tab getting thrown around into everything as it contains the present state of the movie store (who has which movie checked out).
handle_call({checkout, Customer, Movie}, _From, Tab) ->
Response = case ets:lookup(Tab, Customer) of
[] ->
ets:insert(Tab, {Customer, Movie}),
{ok, Movie};
[{Customer, OtherMovie}] ->
{already_checked_out, OtherMovie}
end,
{reply, Response, Tab};
Remember the gen_server:call(....) statements a little bit ago? This is what gen_server calls. You might wonder how there can be two handle_call functions defined with the same arity (the number of arguments, represented by the /x as in "functioname/2"). Look at the first argument, the three element tuple. Now look again at
checkout(Customer, Movie) -> gen_server:call(?MODULE, {checkout, Customer, Movie}).
The first elements match, which is how gen_server knows which handle_call to call to handle...the..call. Waay too many 'handle's and 'call's in that last bit. The important thing is that, like a lot of other things in Erlang, is based on pattern matching.
Notice that the table, Tab, has also been passed in. We go right into a case/of block that checks the table for the customer to see if they already have a movie checked out. If nothing is found ([] ->...) the movie is checked out to them and {ok, moviename} is returned by the block. If they already have a movie checked out, {already_checked_out, moviename} is returned by the block and they are not allowed to check out another movie (note: for this example, there is no way for a customer to return a movie. That is left as an exercise for the reader.).
In either case, the block's return value is stuffed into a tuple and shot back at whoever called the function. Before moving on to the next handle_call(...), notice the last line: {reply, Response, Tab};. You'd probably expect a period and not a semi-colon. The clauses of the handle_call(...) block are semi-colon delimited and will not work if they are terminated like other functions. I found that one out the hard way.
handle_call({lookup, Customer}, _From, Tab) ->
Reply = case ets:lookup(Tab, Customer) of
[{Customer, Movie}] ->
Movie;
[] ->
none
end,
{reply, Reply, Tab}.
This handle_call/3 defines a function to check and see if someone has a movie checked out. If so, it returns a tuple containing the customer's name and the movie title. Otherwise, the word 'none' is returned. I'm not going to spend a lot of time on this since it's so similar to the handle_call/3 we just looked at.
handle_cast(_Msg, State) -> {noreply, State}.
handle_info(_Msg, State) -> {noreply, State}.
terminate(_Reason, _State) -> ok.
code_change(_OldVersion, State, _Extra) -> {ok, State}.
The block of functions that need to be there for gen_server to work, but we're not using this time around. I'll cover these in a later tutorial. A quick note though - code_change/3 is really, really cool. It allows the server's code to be hot-swapped while the server is running. After all, why bother bringing a server down just because the code is changing, right?
Finally, here's how to run it:
2> c(movie_store).
{ok,movie_store}
3> movie_store:start_link().
{ok,<0.42.0>}
4> movie_store:checkout(phil, "Sneakers").
{ok,"Sneakers"}
5> movie_store:lookup(phil).
"Sneakers"
6> movie_store:checkout(phil, "Koyaanisqatsi").
{already_checked_out, "Sneakers"}
7> movie_store:lookup(phil).
"Sneakers"
One problem is that the program doesn't compliment you on your choice of movie if you try to check out 'Koyaanisqatsi'. Well, I've been long-winded enough for one day. I hope this was at the very least, interesting for you. If you improve on this, change it around, or whatever, by all means let me know.
Alright - it turns out the RocketRAID is both a snare and a delusion.
Now of course the OS install disk we're using is informing us that we don't have a valid CD drive attached. Keep in mind that this is coming from a program loaded off a CD. A problem for tomorrow.
On May 9th, I'm leaving the IS department at Rackspace. I like doing dev work on my own, but there is not the place to do it for me. I'm a single Ruby coder surrounded by Python guys and the end result is that I just have nothing to do. I'm looking for something internally, so we'll see how that goes.
It seems that 64-bit Heron has some trouble recognizing RAID arrays. The machine booted and the RAID controller's configuration screen came up. After tweaking the necessary settings, we bounced the box and prayed that Heron would see it - which it didn't.
When we got to the installation section where we were going to do the partitioning, we were given prompted to choose between the twin 250GB Western Digital drives. Thinking that perhaps, somehow, someway the installed OS would see it we continued through the installation only to hit repeated checksum verification errors during the base system install.
Desperately hoping that this was a fluke, we shut off the machine, coated ourselves in honey, and sacrificed our entire apartment complex to the gods of data integrity and MD5 hashing. No luck (which means the install disk was probably corrupt - I'm having a chat with my LightScribe drive once I finish here). Once the honey had been removed and the police had left, we decided that we'd give Fedora a shot instead - the RAID controller specifically says that it plays nicely with it.
So that's where we stand now - even though Fedora 9 is coming out in about two weeks, we can't wait that long. If there are any hardware issues, we have less than two weeks to find them and get the equipment returned. At least for the time being, a Fedora machine is being added to the rack. I used Fedora at the last place I worked and while I didn't have anything specific against it, I didn't feel particularly attached to it.
As long as we're at it, we might as well trade bash in for tcsh - it's about time I learned some C and from what I understand The C SHell is a good place to learn as a lot of the syntax is similar. I don't have any first had experience though so we'll see.
Switching gears, I've found that a good way to gain some basic day-to-day experience with a language is to use it for any shell scripting needs I have. Erlang being my most recent language of study, that's what I'm going to do. If you'd like to give it a shot, Erlang programs can be run outside of the erl VM by typing:
pratzsch@carbon:/home/pratzsch/shell$erl -compile timely_message.erl
pratzsch@carbon:/home/pratzsch/shell$erl -noshell -s timely_message message -s init stop
Excuse me, your forehead's on fire
pratzsch@carbon:/home/pratzsch/shell$
...while it works, I'll probably end up aliasing that set of commands minus the program name to a bash script (oh, the irony) so I don't have to type that novella every time I want to run an Erlang program from the command line.
These are the insides of 'neon', (not quite fully assembled) the latest web server. This is the first machine I've ever had that has hardware RAID. The chip is an 3.0GHz Intel Core 2 Duo of the 45nm variety, also a first.
The motherboard supports both DDR2 and DDR3 RAM, but it was decided that we'd rather have 4GB of DDR2 than 2GB of DDR3. Naturally, having 4GB mandates a 64-bit OS. We figured we'd give Heron a shot and see how that goes.
