Author Archives: crouse

My HP C7000 Saga: Racking the servers

So after I moved the server rack downstairs, not to big of a deal, it was time to get it loaded up… That however is a MUCH bigger deal, since even empty the C7000 chassis weighs in at somewhere around 200 lbs. I recruited Jerry Buckley to help me with this, thankfully, he was available.

This pic is of the rac with the first C7000 chassis in place on the bottom, and the drive jbod next…

Loading Server Rack

Loading Server Rack

This ended up being a poor choice, and in the end, I moved the drive jbod up to the top so it was easier to wire things, as the jbod chassis is shorter in length and would be stuck in between the two C7000’s… not ideal for when you have to reach in there.

In this image, you can see the supervisor checking our work…

supervisor checking work

supervisor checking work

In this picture is the back of the setup, along with the next set of rails for the second C7000, when I just have the one C7000 and the jbod in place… again, that configuration was changed later after I thought about things for a minute or two.

back of server rack 1

back of server rack 1

Below is the first C7000 Chassis with all the interconnects installed.

1st C7000 with all interconnects installed

1st C7000 with all interconnects installed

Next I decided to put the netgear switch in, and again…made a mistake… I thought putting it in the front would be best, but actually, it made MUCH more sense to put it in the back for cable management. So, this isn’t exactly the final setup yet either, and the jbod rails and jbod isn’t put in yet in this one either.

2 C7000's racked

2 C7000’s racked

Final configuration, the back view. I think it turned out pretty well.

Server Rack - Back view - cabling

Server Rack – Back view – cabling

Final configuration, the front view.

Server Rack - Front View

Server Rack – Front View

So far, so good… more later.

My HP C7000 Saga: The 24U rack

Today with Corey’s help (thanks again btw), went and picked up a used Dell 24U server rack. Complete with Front and Back locking doors and both locking side panels. Very heavy duty, going to be a very nice addition for the C7000’s.

Dell 24U Rack - Front

Dell 24U Rack – Front

As you can see in the photo below, it had a few servers in it before, we left the two that were in it, as they were to old to be useful. Ended up keeping the KVM switch, which I won’t need, and when we picked it up, they said they couldn’t find the keys.
Dell 24U Rack front open

Dell 24U Rack front open

We got it to the garage, and as you see in the picture below, Corey took the side panel off, and I spied THE KEYS !!! 🙂 A very nice welcome surprise!

Dell 24U - Side Open

Dell 24U – Side Open

These are the two tags I could find on the rack, not sure if one is a model number or not, still need to check that out.
Tag - Dell 24U

Tag – Dell 24U

Tag - Dell 24U

Tag – Dell 24U

Only thing left to do… get it downstairs… so, I wonder which son/son-in-law/future-son-in-law I’m going to have to bribe… 🙂 🙂

My HP C7000 Saga: 2 chassis aren’t enough, 3 is better.

Ok, so I’m sure my wife thinks I’ve gone crazy by now, but she smiles and doesn’t judge, what a woman. I of course bought 2 C7000 chassis fully loaded with 32 blades in March, but one of the chassis was an older one. Now this doesn’t REALLY matter as you can upgrade the firmware on it, and other than the looks, there really isn’t much difference. That’s one of the awesome features of the HP C7000 platform, the chassis works the same, and runs the same stuff, that part doesn’t change much over the years. I found however a deal I just couldn’t pass up. I found another C7000 chassis. No power supplies or fans, or admin modules, just the chassis. The price was right, so I took a day off of work, and hopped into the Escape and off I went to Blue Springs, Missouri to pick up another Chassis. I was pleasantly surprised at the condition of this 3rd chassis. It was “like new”, and didn’t appear to have anything wrong with it, in fact, I’m not 100% sure it ever has been used at all. Other than a little dust on it, it was in PERFECT shape. I almost felt guilty talking them down 50% after I went and picked it up…



As you can see, not a scratch on it… so, you ask, WHY did I buy this third chassis ? Well, I wanted two of them to match exactly, and well… it was just simply too good a deal to pass up, my final cost $75.

My HP C7000 Saga begins.

Two C7000 Chassis and Blades

Two C7000 Chassis and Blades

Back in the middle of March of this year (2015), I bought one C7000 Chassis with 16 blades in it, a friend of mine bought one as well. We traveled to St. Louis, Missouri and picked them up. This is the picture of them in my Escape after they were loaded up.
After loading them up and getting them back to my house it was very late, so he drove home and I went to get some much needed sleep after 10+ hours of driving. After a few days, we came to an agreement, I’d give him my Sun SunFire X4600m2 server for his blades/chassis.

So, now I’m the proud owner of 2 blade chassis and 32 blades… 🙂

The blades consist of the following:
[Dual CPU] 4 core Intel Xeon CPU’s E5450 @ 3.00GHz (32GB Ram) (Benchmark 7,796) – 7 -G5 blades
[Dual CPU] 4 core Intel Xeon CPU’s E5450 @ 3.00GHz ( 4GB Ram) (Benchmark 7,796) – 1 -G5 blade
[Dual CPU] 4 core Intel Xeon CPU’s X5355 @ 2.67GHz (16GB Ram) (Benchmark 5,890) – 24 -G1 blades

The G1’s use PC2-5300F-555-11 memory as pictured below:

2GB 2Rx4 PC2-5300F-555-11 Memory

2GB 2Rx4 PC2-5300F-555-11 Memory

This is a picture of the inside of one of the G1 blades.

Inside a G1 Blade

Inside a G1 Blade

This is the information on the top cover of the G1 Blades.

Cover of G1 Blade

Cover of G1 Blade

Now I own two chassis worth of blades, and of course that means I have two chassis, as pictured below. The first one is the one that has all the G1 blades in it, and it has older onboard administrator modules, they don’t have the blue kvm vga adapter that the other one does. I may or may not switch those out. It also has two 16 port ethernet interconnects and not shown, a SAS interconnect for a jbod.

Chassis 1 Back Image. G1 Blade Chassis.

Chassis 1 Back Image. G1 Blade Chassis.

This second image is the back of the blade chassis that contains the G5 blades. It has new OA modules for administration, as shown by the blue vga kvm ports.
This also has the SAS interconnect for Jbods. This shows all 10 fans, and the 6 connections for power (110 OR 220 work). In this image I’m only using 3 of the powercords as they are connecting to one 20amp outlet, and this will power only 7 blades total before it degrades the chassis and it will refuse to power on more.

Chassis 2 back image. G5 blade chassis.

Chassis 2 back image. G5 blade chassis.

This is the front of one of the chassis after installing some 72 GIG hard drives (raided) used for installing the OS (bare metal hypervisiors actually).
Only powered on 2 units to make sure they would go green, the yellow one is powered down, the yellow is the normal color you would see when in this state.

Inital poweron of a few blades.

Inital poweron of a few blades.

When you first power on an HP7000 you can login to the primary OA “Onboard Administrator” through a web interface and run a “First Time Setup Wizard”. Worked pretty well, I ended up making a ton of changes after that though, but this is what that looks like when you get there.

HP C7000 "First Time Wizard"

HP C7000 “First Time Wizard”

Below you can see I have only 4 blades powered on/installed. This is because I wasn’t sure at the time on how many blades my single 20 amp outlet could power, now I know it’s 7 blades.
Shown in this image is the “Device Bay Summary”, which is where you go to see your blades status, and to quickly login via web interface to the blades ilom, by clicking that blue “Ilo IP Address”.

HP C7000 OA "Device Bay Summary" Page.

HP C7000 OA “Device Bay Summary” Page.

Below is the image of the “Power Subsystem” page that shows what 4 blades and the Chassis are using while I was working on them.

power subsystem  - usage image

power subsystem – usage image

blue ilom light image

blue ilom light image

The image above shows the 4 blades and one of them has a blue light on, that indicates that someone is logged into it via the ilom. You can manually turn that blue light on and off if you wanted someone to pull a blade at a remote site, that would be a good way to indicate for sure which one you were wanting removed/etc.

Well, that’s it for now. I’ve gotten alot further than this, but I don’t have images to go with the updates yet… stay tuned, I’ll post more on the updates and progress I’ve made soon.

The SUN SUNFIRE X4600 M2 Server

Every once in awhile I get to go crazy and build something fun at home. This time it’s a new virtual machine setup, as my older one only has 32 GB ram and is getting pretty close to it’s limits. So, time to build something bigger, and cheaper, than I currently have. Plus, it’s a good way to play with new virtualization technologies.

So… I am now the proud owner of a SUN SUNFIRE X4600 M2 Server.


Specs are as follows:

Model: SUN SUNFIRE X4600 M2
CPUs: 8 qty: (AMD Opteron 8384 Quad Core – 2700 MHz – 32 cores total).
RAM: 176 GB’s Ram. (44 qty: 4GB PC2-5300P DDR2-667 RAM ECC REG CL5 Samsung)
DRIVES: 4 qty: 146GB SAS
WEIGHT: 95 Lbs.


Currently, there is only 172 GB of ram installed on the machine, I hope to get it to it’s max of 256 GB of ram within the next week or so.

It was an interesting thing getting this running well, and while others may not ever need the information I provide, I will share what I have learned about the machine. This machine has 8 “8-DIMM Split Plane CPU Modules” in it. In essence, mini-blades that all look like ONE machine and not 8 when turned on. Sounds cool, but it wasn’t as easy as just turning it on of course.


I was lucky enough to have a friend named Mike help me get some of the bugs worked out. The first bug we encountered was that the machine was seeing only 2 of the 8 modules. So it turns out that the x4600m2 is rather picky about things… every two modules had to be exactly alike, and the ram in each of those two modules had to MATCH in pairs. After pulling all the modules out, and re-arranging the ram to be exactly the same in pairs of modules and matching dimms by pairs in paired modules… things got more interesting.

Still, it wasn’t seeing everything correctly… So on the far right side on the back of the box is a pinhole to reset the cpu’s… pushing that in with a paperclip while restarting the machine made it see everything just right finally.

The second issue we ran into was that older iloms don’t tend to work well with newer browsers, and neither Oracle or the browsers care to really fix that issue, since it’s older equipment. I did figure out how to get past that and wrote about that here: How to make newer firefox browsers work with older Sun iloms. With that fixed, it’s now time to play with the machine more and see what other issues I can find.

The third and final issue for the month however might end up being my power bill…. LOL 🙂

All in all, I am pretty happy with the purchase. By the time I max out the ram at 256 GB I will have a grand total of about $500 in the setup. Not bad for a virtual machine host, 32 cores at 2.7 Ghz and 256 GB ram… I should be able to create a “few” vm’s in the test environment with it I think. 😉

How to make newer firefox browsers work with older Sun iloms.


Found this the other day when trying to get my X4600 M2 Ilom to show up right in my web browser. Basically all the data was missing from the tables etc. (Note: I’m using Firefox 26.0, on Linux) — works on Windows as well. Neither Oracle, nor Firefox really care about making things work on the newer browsers with the older iloms…so this might just be the easiest fix.

This workaround on a per user basis works;
Add the following 4 lines to the file (create it if it does not exist) ~/.mozilla/firefox/*.default/chrome/userContent.css and restart browser:

@media print {
@namespace url(;
#mainpage { visibility: visible !important; }

Why ZFS is so cool…

So, I attempted an upgrade while not at the servers…but remotely. So I didn’t take a snapshot with vmware, thinking it would be a minor deal. Needless to say, the upgrade failed horribly, and I was really wishing about then that I had tried this while in front of the server AFTER taking a snapshot…. That IS my normal course of action, but alas, not today.

So, I logged into my internal network and proceeded to login to my NAS and my VMWARE ESXI host via the command line.

First order of business, turn off the VM on the esxi host…

I have to find out which number the vm is so i can turn it off:
vim-cmd vmsvc/getallvms

Now, I see the number of the vm beside the name, so…duh…turn it off… 🙂
vim-cmd vmsvc/ 53

Next I need to restore my data back to a previous state in my NAS…
Lets move the borked vm data out of the way…
nas4free: ~# cd /mnt/NAS/virtual_machines/
nas4free: virtual_machines # mv ws-crouse ws-crouse.bak

So, now, to find out what I have for snapshots that NAS4FREE takes automatically !
nas4free: # cd /mnt/NAS/virtual_machines/.zfs/auto-20150123-000000
nas4free: auto-20150123-000000 # ls
nas4free: auto-20150123-000000 # ws-crouse/

TADA … I have snapshot… now to copy it back !
cp -p -R ws-crouse ../../../

Back to the VMWARE esxi server and turn this puppy back on!!
vim-cmd vmsvc/power.on 53

All this in less than 5 minutes.
Gotta love the power of zfs snapshots!!! 🙂

Scrivener – Your complete writing studio



“Scrivener is a powerful content-generation tool for writers that allows you to concentrate on composing and structuring long and difficult documents. While it gives you complete control of the formatting, its focus is on helping you get to the end of that awkward first draft.”

They now have a Linux (beta… but actually runs REALLY well). Used by authors and novelists, it is a pretty cool way to start writing that first book !

Download here (now 32bit and 64bit versions available):

Learn the Bash shell.

The poor Bash shell script. It gets lost in the flood of all the glitzy language wars, Python vs Ruby vs Perl vs Javascript… the list goes on and on, with new contenders seemingly rising from nowhere to get in the fight for developers attentions. Seldom do I hear anyone stick up for the poor little Bash script. Seems like everyone likes the glitzier languages more. There are books galore for all the “big guns” of the programming world, and more of those seem to pop up everyday, seldom do I see a new book for Bash or Bash scripting show up anywhere.

I know I’m a bit partial to Bash, being a Systems Administrator, I use it daily. Almost a decade ago, I started a website, because I really enjoyed the language and there seemed to be no end to all the cool stuff it could do. So, I’m here to stick up for this under rated programming language. Yes, some of you will look down on me for calling it that. But it is of course a programming language. Concise Encyclopedia defines a programming language as a: “Language in which a computer programmer writes instructions for a computer to execute.” So, you see, you are programming, regardless of how others try to redefine it. Don’t let anyone kid you, you ARE programming. Some people prefer to call it “scripting”. Like that’s a bad thing, and beneath them. HA. They know not what they are saying. Learn the Bash shell and the gnu tools that most distribution install by default, and you will always have a Swiss army knife of knowledge for working with your operating system. Be careful, or you might also get introduced to that nasty Mr. Awk and Ms. Sed as well. If your new to Linux, by all means, make learning Bash scripting one of the top priorities on your list of things to learn.

When writing shell scripts, you will probably also learn to include outside programs, like curl, ssh, and other useful tools. Learn about all these cool tools that people before you have built and made available for you. Bash is the super-glue that can bind these powerful programs together to build even more powerful programs. Not so far removed or different from say, Perl modules, or Python modules. If you need to, you can even include entire Perl or Python programs IN your shell script. I often see opinions like, “Bash is only good for those one liners, small stuff. Anything very complicated should use (insert language and flame war here)”. They simply haven’t seen any of my 2,000+ line Bash scripts, or what they can do. One of those “scripts” happens to be a cluster controller for 4 Oracle T4’s running Solaris each with multiple zones that could turn on and off those zones and move them from machine to machine and could give you a representation of what each had running on it, in color no less. One of the other administrators where I work, demonstrated it to some Oracle engineers that were visiting one day, they gave it the thumbs up. Please don’t let anyone fool you into thinking that learning the Bash shell, and Bash shell scripting, is a waste of time.