Life in the Data Center

Microsoft Azure – Life in the Cloud

I had a chance to view the Microsoft Azure platform last night.  It was presented by the DC DotNet Users Group in Washington DC.  I am quite familiar with both Google and Amazon cloud’s and the advantages, and disadvantages with them.  For my enterprise, they are not very helpful as we are a 100% Microsoft shop and run tons of apps, both off the shelf, and custom .asp/.net.  So most clouds are difficult for us for the following reasons.

1. Can not install .exe or .msi apps and run out of the box.  Most clouds require some proprietary process for getting apps created and uploaded to the cloud.  Kinda like creating an image for a desktop/laptop.  There is a process to follow and you have rules to follow or you end up with a messed up image.
2. Because of the proprietary nature, it requires changes in internal processes, additional training, and plain just changing the way you do stuff.  It also requires you to conform to the cloud and you possibly loose flexibility.
3. Linux – Open source based.  We are a Microsoft shop, so the application-development team would have to switch or create and maintain two code bases.  Neither is attractive for us.

Now comes Azure from Microsoft.  It has been out since January and building some steam.  Here are the pros!

1. Its all Microsoft Stuff.  Yeah.
2. No need to procure Microsoft licenses.  Its all built into the costs.
3. Developers will like it.  Create apps for in-house or the cloud.  If you do it right, you can deploy either.
4. Typical cloud.  Spin up virtual servers pretty quickly.  Networking, firewalls, load balancing, etc all built in.
5. Geo-location.  You can scale to multiple data centers pretty easily.  Just costs more.
6. TCO is ok.  Scalability is key.  Just depends on your requirements.  No CAPEX.  All Expense!

Now the cons.

1. While its all Microsoft software.  Its not the same.  There are lots of limitations.  You can not tweak most of the stuff you want to tweak.
2. No installing .exe or .msi’s like you do on a standalone boxes or VM’s.
3. No access to IIS configs.  No access to registry.  No access to boot drives.
4. Azure VM boxes are throw away’s.  You need to store code and temp stuff on some type of storage.  So if you do certain things now, you have to learn the cloud way.  This is likely to be painful the first few deployments, upgrades, mistakes, etc.  I have it down now, but now I need to learn the cloud way.  Its new and you will need to learn it.
5. Costs.  Can get expensive.  You pay for everything.  Development, staging, production environments all cost $$ as they need to be running.  So if you run a typical three tier environment, it all adds up, and that erodes the cloud TCO compared to running in-house.

It would be tough for us to use Azure at this time.  We use a lot of off the shelf applications and tools.  No way to install into the cloud.  Applications want to install and modify the local drives, copy files, change the registry, and all that is not cloud compatible.  I am sure if we could redo some apps, start all over with the latest development tools and develop for the cloud, then that may be an option, but it would require a wholesale redo of an application.  Given available resources, its not likely, nor is it cost effective.

Why?  Because we do data center stuff pretty good and we concentrate on availability.  The big advantage is DR/BCP.  The cloud could certainly could help us there.  But our largest web apps are not suitable for the cloud, because they are built on a third party applications, so its not really an option.  There are some suggestions for putting Exchange and IM in the cloud and somehow reducing CAPEX costs.  Not really!  We do it pretty good and we also integrate Exchange and IM with voicemail, Blackberry’s, and mobile devices.  Shortly we will step it up and integrate with our Cisco phone and Tandberg video platforms.  So outsourcing Exchange and IM to the cloud, would be painful given our integration points, and certainly make it more complex and expensive.

My $.02 on the Azure cloud.  But it will change and improve over time.  I will certainly keep an eye on it.  Ts-Admin.

Initializing your CX4 Array

So you got your CX4 cabled, power attached, and ready to start up.  Go ahead and flip on each SPS.  Let it start and give it plenty of time.  Be sure each SP’s light is steady green and there are there are no amber or red lights blinking.

Now find the “Navisphere Storage System Initialization Utility” CD.  Initializing allow you to access the array from the networking using a web browser.  It may be on the server support CD or may be the CD that says “install me first”.  Follow the instructions that say install products on server.

1. Connect each SP to a very basic network switch.  Nothing special, just a 10/100/ switch, and connect your laptop to it.  Give your laptop an IP address.  Anyone will do.  The array and laptop need to be on the same physical subnet.
2. Start the Initialization Utility: Start, Programs, EMC, Navisphere, Navaisphere Storage System Initialization.
3. The utility will automatically search for uninitialized storage systems.  They are identified by serial number.  Check the back of your CX4 and verify.  Especially if you did not do #1 and have it connected to your network.
4. Select the serial number from the Uninitialized Systems list.  Provide the information the utility requests.  It is all IP address stuff.  Use the IP addresses that you want to use once you have it connected to your network.
5. It likely will want to reboot the array.  Let it.
6. Connect the array and your laptop to your network.  Update the IP on your laptop if necessary.  Pull up a browser and browse to the IP address of SPA.  You should get Navisphere login.  Go a head and browse to SPB.  You should get a Navisphere login.
7. Register your array!

BAM!  Your ready for the next steps.  Which is to update the array with any patches and/or enable additional software.   More on both in a later post.

Just Added Social Media Plugin

Please feel free to spread the word on posts that may be helpful to anyone. Or may just be useful or knowledgeable for someone! Thx, Ts-Admin.

Upgrading to WordPress 3.0

Upgraded the blog to WordPress (WP)3.0 this weekend.  Overall it was pretty easy, and since it is a VM, it would have been quite easy to revert back and start all over.  Because WP keeps almost everything in a DB, it really involves replacing the html code, and running a script to tweak the DB.  Below is a quick procedure.  It is for v2.7+ and the upgrade guide is here.  Here is what I did.

1. Shutdown the box and took a snapshot from the VM Client.  This allows me to fall back to this point in time.
2. Boot the server.
3. Using a web browser, I backed up the database.  WP uses MYSQL.  So I logged on to MYSQL via PHPMYADMIN.  This is an optional application and makes management simple.  From there I backed it up to my desktop.
4. I accessed the blog admin page and disabled all plugin’s.  This is recommended.
5. I downloaded the wp3.0 file and moved the file to the server.  I unzipped to a temporary directory.  I also created an archive of the existing installation.  I now have everything I need to also fall back another way.  I can copy the existing install back and restore the DB.  Or I can use VM to fall back in time.
6. I followed the guide and deleted most files in the root of the site and also some folders.  I left a few files in place.  Again, the guide provides specific instructions.
7. I copied the WP files to the roof of my web server.
8. I ran the DB upgrade script.  Success!
9. I accessed the blog and checked the admin site.  All is good.
10. I activated the plugins.
11. Two plugins needed upgrading.  I upgraded and tested.
12. While I was here, I added a social network plugin.  I tested and its all good.
13. Finally, updated the server OS with a few application upgrades and patches.
14. Rebooted.  Tested.  Done.

I will run it for a week and then remove the VM snapshot.  It will also get backed up as a complete VM by one of our backup servers.  I can now restore this as a new VM at anytime and restore the entire site within an hour.  VM’s are great for this.

DotNetNuke Hack – Update/Check your Stuff!

The challenge with running a data center is you generally have lots of stuff, both equipment and software, making it all work.  Add in 3rd party tools and developer applications, and its a challenge keeping it all running.  But hey – That’s our gig – We make it run.  Sure these tools make the developers life easier, so they do not reinvent the wheel all the time, but when you rely on third party apps, you inherit their vulnerabilities, and that is a bad thing.

DotNetNuke (DDN) is an example.  DotNetNuke bills itself as the leading web content management platform for .net. Well you better check your stuff, because it can be hacked pretty easily if conditions are right.  See http://blog.aggregatedintelligence.com/2010/02/dotnetnuke-version-zero-day.html for an excellent example of the exploit.

So did we get hit.  Yup.  On a in house developed blog system and it is a deface hack.  What did we do?

• Site was attacked using a vulnerability in DotNetNuke
• Attacker used the vulnerability to drop a script and deface all sites on the server
• Once discovered, server was immediately taken offline and recovered.  The server was virtual and easy to take off line and still continue to access and identify the issue.  VM’s are great in these situations.  You can take snapshots and duplicate for a variety of uses.
• Appropriate measures were taken to prevent the attack from happening again.  See below.
• Server was brought back online.
• Remote attacker IP indicates it was a hacker from Iran (IP Address: 94.183.194.10)

This was pretty easy to resolve.  Once we had the box offline, we did the following.

• Deleting all files that were modified or inserted
• Modified the vulnerable file so it could not be accessed again
• Modify permissions on the web site files to deny writing files
• Modified registry permissions to deny access to the scripting object that was used to write the files
• Repaired any missing or altered files for critical applications

We also reviewed the use of DDN in all other systems and went back to mitigate them as well.  If you use DDN, be sure your running v4.9.4 and up, or you may be vulnerable.  And its a pretty well known vulnerability.  Just Google “|| IMAN_TAKTAZ WAS HERE ! ||” and you’ll see tons of references.  This was the deface.

The awesome part of this?  We had it identified with 15 minutes of the hack, brought the site down, recovered the site to a maintenance page, and then began our internal process of figuring out what happened.  This took some time and we were not worried out time anyway, but we were worried about the hack and the possibility of a compromise.  So since we had a maintenance page up, we brought a few experts together on a phone bridge and started sorting it all out.  You really need the developers and web folks there to sort through the logs and interpret what occurred.  We have sharp dev-app guys and they parsed it all and figured it out.  Then we had plan for mitigating and we got it back in production.  Sure it took a few hours, but we are a careful lot.

How did we catch it?  Our monitoring system!  It monitors for content.  So when the main page was changed, it tossed us all an email.  Then we sprang into action.  And we were lucky!  Always helps if your lucky!