Hold On

They're adding a second to 2008. You'll have to postpone the celebration for the specified time period. Of course, you won't know unless you have one of these.

I do. Are you ready?

Prone to Wander

Males Dominated 'Out-of-Africa' Migration - Discovery.com

I have always wanted to move, no matter where I was. I didn't like it anywhere, and I still don't. Today I read that it might just be that I'm a man, and men want to go. Maybe I'm not alone.

There's a lot I want to do in this life that I have not done. I'm rather upset that I haven't gotten around to doing it, and the frustration sometimes consumes me. And then I'm lost again - I give up and want to sell all my books and throw everything away. Streamline life and live simply. Pick just a couple of things that I say that I do, and forget everything else.

But I can't. Every time I think about shedding my interests I find it impossible to do so. I am the guy with too many hobbies. Can we count?

1. Education - I love to learn. This is the catalyst of my affliction.
2. Family. It's growing.
   
3. Home brewing. I love beer.
4. Computers. My living. Could be broken down into many smaller categories.
   
5. Motorcycling.
6. Business.
7. Pleasure reading.
8. Periodicals (deserves its own listing).
9. Movies.
10. Gardening.
11. Model airplanes.
12. Flying (one of those things I haven't gotten 'round to).
13. Music. This is big.
14. Writing.
15. Traveling.
16. Language.
17. War.

That last one is something new. I have recently become interested in joining the military once more as an officer, and if one were to choose the Army as a career, an officer in the Army would need to know about strategies and history related to war. Also about weapons and how to use them. So I want to study war.

But I won't get around to all of this and it is very frustrating. I have so many interests and I am forced to choose. Does anyone have a suggestion as to what I can do?


Cross-posted at the WordPress blog.

Starbucks is Lame

I’m amazed at the uncharacteristic way that Starbucks contributes to its community. The coffee is good (I will not deny), but the fact that everything is expensive really detracts from the homeliness of it. How does Starbucks let Internet providers charge for access? Shouldn’t the damn place just offer it for free? Does it not want a crowd?

I’m writing this in Starbucks tonight (not online) and as I write I wonder why I decided to stop in and have a coffee. I really wanted to get an assignment done. My school is online and the thought occurs to me now that I might have been more productive at Wendy’s.

I had a plan last year to gain Internet access from Starbucks. I offered Fazoli’s next door a great deal on a WiFi hotspot, which I could simply enhance so that I could reach it from Starbucks. I hate Fazoli’s. The owner, after months of deliberation, decided against it at the time but reserved his right to call back if he changed his mind.

I just spilled coffee on the notebook I carry every day. Gives it character.

So why did I stop here tonight? I guess it’s the fact that I could have sworn that AT&T was offering free WiFi at Starbucks. Turns out that $3.99 is not the same. $3.99 for two hours of use. That’s worse than T-Mobile’s $9.99 per day. The subscription is the same at $19.99 per month, but who wants to pay monthly for two Internet connections? I’d rather pay $60/month for a WWAN card and not have to search for an AT&T hotspot everywhere I go.

I remember when I got my first printer. It was an all-in-one from Compaq that eventually turned to shit, but it was a printer. And I remember after a few months wondering what I did with a computer when I didn’t have a printer. “What good is it,” I asked, “if I can’t print anything?” I now have nixed that idea and replaced it with “What good is a computer without the Internet? What is it for?”

But somehow I’ve managed to keep this MacBook open at Starbucks without hitting the ‘net. Wait a minute… Wendy’s is close… Nevermind.

I’m typing in Word, and I don’t have a printer or a connection to the Internet. Disconnection is great! But how do I play Mob Wars on Facebook or update my status?

That’s it – I’m opening my notebook.

Crossroads

I can't seem to have enough drama in my life. I'm always moving and changing my mind and even now that I have decided to do something, I can't decide which way to go about doing it.

For those who don't know, I have decided to try to enter the military again as an officer. I thought I wanted to join the Air Force (and some part of me does), but then upon finding my old company commander for a recommendation I was spun toward the Army. Beth would like to see me join the Air Force if I just have to go, but there's a sense of pride about being an Army officer that I cannot explain. I am torn at the moment.

My friend Ken said I should do a trade-off analysis. I'm sure he was talking about the trades between civilian life and being a soldier, but I can compare and contrast the two branches too. I can even do some of that right now:

The Air Force is more technical, hands down. I belong in information systems and everyone knows it. There are nicer things, I've heard, in the Air Force. I think the physical fitness tests are taken on stationary bicycles or something. I love aircraft. But there's no guarantee that I would work with computers as a career.

The Army is more... well, military-like. Officers lead soldiers and make strategic decisions. It's not really fun work but it is something to be proud of. There is a higher chance of losing a limb in the desert if I choose to go to the Army. It is more physically demanding and I was never any good at running. Not that I can't be, but I wasn't. But like the Air Force, there is no guarantee that I would get near a computer. I believe the chances are even smaller here.

Ken would like to see me as an Army officer and I respect his opinions. He also told me that I wouldn't get the opportunity to lead in the Air Force. I should confirm that. Beth hates the Army and thinks the Air Force is better for the family. After everyone says everything, it's really my decision and it should be made based on what I want to do for myself, my interests, and my family. It's just a big storm in my head and it may be a long time.

I started writing the letter required for the packet that states why I want to be an officer in the Air Force/Army. I haven't finished, and the letter may be different for each branch. I know that I want to serve this country and use my skills for something good. I know I can make decisions based on facts. I know I can lead. I just don't know if the green pill will take me down a better trip than the blue one.

This is one of those rare times I wish I could pray.

This Blog is Going TDY

I am moving - at least temporarily - to a new blog at WordPress. I doubt it's a permanent move, but I want to see how different the free WordPress is from the free Blogger. I have already found out that I cannot publish the WordPress blog here at benrehberg.com without paying a premium, so that's already a detractor.

But the theme is great and I want to find a better one for this site. And I like the interface at WordPress. So if this post gets old, check over there to see if I'm still writing.

RAGBRAI 2010

I was thinking about my future today, and I have come to the conclusion that I can't switch jobs for another two years. Here's why:

I have said I'd go to RAGBRAI in 2010 with Scott (and he better still be on board), so I can't start a new job because then I won't have the vacation time saved up.

And that's all. That is the sole reason I can't move or change jobs. Things like that can really screw up some plans.

Oh, and I just thought of this - after five years at this job, I'll be vested in the retirement plan. Not that it's worth anything, but I might get some more out of it than I would by leaving early. I don't think I've ever stayed at a job for more than four years. Next October will be four years at this one, so we'll see. I hope it gets a little more interesting. But that's for another post.

Can't Say I Didn't Think About It

I've spent some time with my neighbor recently, and discovered that I am slightly envious of him. He lives alone, and is a senior NCO in the Air Force. He is two years older than I am, and he has six years to go to retirement from the military. He seems to have everything going for him and all is in order. Oh, to be like that.

I have battled for several months now about joining the military once more. Sixteen years won't last that long, especially if I choose to enjoy my time. The four years I spent in the Army were very long because I resisted liking anything about where I was or what I did. I want to change that and retire in sixteen years, and then go get a job so I can retire again.

But some things make me not want to. First: the pay. I think I make more now than I would if I went back in, and raises would not come so frequently. Second: I'd have to travel some, for extended periods of time. I have become comfortable being home, especially now that we have a child. When I am gone, I can at least call and be called by the family.

But how else can I get a chance to live in Germany? To have free health care for my family? To go to the gym for absolutely free, and probably find someone to play racquetball with? To have a structured daily life? To not wonder what to wear? To get real-world experience with computers, a security clearance, and probably a $100K-plus job after retirement?

So I'm stuck. It's stability versus comfort. I don't think I'll ever stop wanting my life to be different.

Really Beta.

I got an e-mail from Apple yesterday about joining the new iPhone/iPod developers' forum, now in beta. As an iPhone developer, I have access.

The system runs on Apple servers, running Java, serving JSPs. I had an error setting up my account this morning, and it looks like the whole thing broke. Maybe they should set up a forum to discuss forum issues.

Business Unusual

I got a call today from a woman who works in an office I support. She wanted to buy a laptop tonight. And then she showed up and bought one.

After all the boredom and worry about that little computer store, someone manages to come in and buy something every month. Maybe I will actually make it.

Impatience

I've been impatient. That's the root cause.

I have had this familiar fruity taste in all the beer that has come out of my Mr. Beer kit, and I discovered tonight just what that peculiar taste was: wort. Unfermented beer. I hadn't waited long enough for the yeast to finish consuming all the sugars, and there I was, drinking unfinished beer.

It is very apparent in the latest batch of brown ale, and I had to even pour out a glass. I bottled it after seven days, but should have gone longer since I added Sugar in the Raw during the boil. I took it all out of the refrigerator tonight and will let it sit on the counter for a week. I just hope the tops don't blow off.

The blonde ale I have in the fermenter will stay much longer. Today is the 9th day in there and I don't have any intention of bottling it this week. I'm so disappointed in the last brown ale that I'm going to sit on this blonde for a good while. It's still bubbling, and I hope it turns out okay. Two more ingredient kits are on the way from Mr. Beer and my five-gallon equipment has been shipped. I might need to borrow a refrigerator or have a huge party.

The Debate

My Girl

Anika is a fantastic child. I just thought I'd share with the world this morning's joy.

Kids, You Know?

I went to a meeting Tuesday night with a handful of local merchants in Cook County. We're (they're) trying to get a group together to help bolster small business revenue locally. I took Anika with me since Beth has school Tuesday nights. Anika loves my computer shop and even has a cot there to lay down on.

The mayor was at the meeting, as were the owner of the local newspaper and several other business owners (obviously). I had a hard time keeping Anika quiet and to herself as she grew bored. Once about every 150 seconds, I tried to tell her to quiet down, quit kicking the table, don't draw on that, etc. There were several other conversations going on in addition to the speaker of the meeting. Anika wasn't the only person bored enough to find other entertainment.

I had to get stern with her one more time, and turned around to listen (or act as if I was). I felt a pat on my arm and turned back to her. She was pointing directly at Mayor Barr and said aloud, "Well he needs to be quiet too!"

I love her. She's so fair.

No One

"Never underestimate the disparity between developer excitement and user apathy."
-Ted Dziuba

I was reading this article this morning and came across that line. It's a good one-liner that encompasses a lot of what I feel sometimes.

I read the tech magazines and have my cool 2600 shirts, but no one can relate. Nearly everyone I know has better things to think about, and they don't give a shit about what it is I do. That is, until they have a problem with some electronic device. But even then, they just want it fixed. They don't really care what the problem was. They ask, but they don't listen to the answer or really care for my explanation.

I like to think I know what I'm doing most of the time, and so far it's working. It's just that I'm lonely in my enthusiasm about computing. There is no one here like me.

Let Me Say This

This is in response to this little post. I'm not opposed to helping people. Yes, I'd like to pick the ones I help (who wouldn't like to know where their time and money are going?). The rich get richer and the poor get poorer because too many people think that they can't change. I'd like to imagine that if my house was flooded and everything was gone, I could make something somewhere else and get through. I wouldn't be cursing the government or the rich because I was hungry. I don't know a single wealthy person who would refuse to grant me a meal or even a night in their house. Some folks I know even have a spare house or two I could probably use.

As a small business owner, how am I to respond to that article? I am certainly toward the bottom of that pyramid mentioned, but I like to think I make good decisions based on what I know. It's only my fault that I am where I am, whether I'm at the top or the bottom. Should I make more money and hire someone to pay, or should I just try to make rich people poorer to equalize the social status in my region? I don't get it. What is the purpose of starting my own business if it is so wrong to build wealth and pay poor people to work for me? I probably couldn't find any wealthy people to sit in my shop - I'd rather pay someone who needs the money and wants the experience.

This is all difficult to explain, but I just don't get the point of the article. Maybe it doesn't make a point. It offers no solution. How am I to respond?

How I'm Gonna Feel

Today Begins...

I was supposed to start doing something at 8:00 this morning, but now it's after 9:00 and I've yet to peel my buttocks off this chair. I actually have a full calendar today, but I just don't feel like doing much.

Now that you've helped me think about this, I'll go to work.

I Should Live in Canada

I was going to post a cute little thing about one of those personality quizzes I took this morning, but the code to embed it is really bad and it clashed with Blogger's editor anyway.

So I'll just tell you about it.

No I won't. It's not important. It was a waste of 30 minutes, which in turn made me waste your two minutes reading this post. And you're no better for it. Some friend I am, huh?

Wish You Were Here

Looks like Friday will be rough. I expected to see storms by yesterday, but Fay decided to take another trip to the Atlantic and get her wits about her (you know she's tired after touring Florida). Then she's coming over. It's warm out in the Atlantic and some say she could become a hurricane before she comes to Georgia. The tip of the yellow arrow is about where I live:

Another Idea Down the Drain

Well, there you have it - right there in the agreement for the iPhone developer program: You can't drive a cruise missile via an iPhone application. There goes my fancy new government contract. From the agreement itself:

Applications may not be designed or marketed for real time route guidance; automatic or autonomous control of vehicles, aircraft, or other mechanical devices; dispatch or fleet management; or emergency or life-saving purposes.

Doesn't that suck? Just what exactly are we to do with our iPhones* if we can't blow something up?


*I don't own an iPhone and plan to only if I develop something really cool that needs testing.

Learning a Lesson

I know I shouldn't hire anyone. I'm stressed to be efficient so as to reach a point at which the company is profitable, and that means I'm in it alone for a long while.

But I really have to poop.

And I'm sure there's some law that says as soon as I get back there and down to business, a customer will walk in the door. I haven't had a genuine customer just walk in yet, but if I go back there and get to it, I am sure I will meet my first.

In good spirits,

Reach for the Sky

We've had a hold-up. My day job announced that they need me tomorrow through Friday in Atlanta, and I won't be able to operate the store from there.

I have given some thought to setting up the iMac on the counter and doing a video chat with customers as they walk in, but that would be more of a pain right now. Maybe when I get more established downtown I can do a remote-consultation type of thing from afar.

We will now be opening the storefront on August 9, 2008. We'll see how that goes.

It's On Now, Bitches!

This is only the beginning. From this photo, we're much farther along than this today. We should be open for business next weekend; I'm shooting for Saturday, August 2.

Rehberg Technology, Inc.

Let's see... what did I get done today?

I woke up, worked in the shop several hours, came home, drank some beer, ate dinner, posted to a class forum, and then I formed a corporation.

Who thought it would be that easy? Just a quick web form and $100, and I have myself a business entity. Rehberg Technology, Inc. is as of right now Georgia's newest corporation, and I couldn't be more proud for them to name me as the CEO.

Which reminds me - I have to stop by the paper and give them $40 and the instructions for the press release (notice of incorporation).

It's late now and I'm spent. I formed a whole company, for chrissake! Good night.

Mr. Dvorak, You Are Wrong Again.

He thinks that Internet access should be metered, just like the power and water. You pay for what you use. His argument is rational, I'll admit, but it is not inclusive of every use.

I'm writing in a hurry because I really want to discuss this, but Beth won't let me take a computer this weekend to Florida. I'll put the link here so you can take a look, and I will discuss this at length at a later time. I'm shutting down now.

Changed My Mind About iPod Touch

I saw an ad on Gmail from Amazon, pushing the iPod Touch at a discount. 9% off the 32GB, bringing it down to $460 or so. I wanted to look at it again, so I went to the Apple store and watched the videos. Some of the features are great, but they're just not worth $460. For instance, any PDA these days can play music and videos, and most of them in that price range are equipped with Bluetooth. The iPhone with a contract has Internet access everywhere. Not so with the iPod.

No Bluetooth, WiFi only, no GPS (seriously, how hard would that be?), no replaceable battery, and it's $500. It just doesn't make sense. I'm going to wait for a multi-touch enabled Open Handset sporting Android. From the emulator that comes with the SDK, I expect to have everything that could possibly be crammed into a handheld device at my disposal. Bluetooth, WiFi, 3G, GPS, accelerometer, multi-touch, and possibly a keyboard. All on an open platform for which the API is very open. Sure, I won't have 32GB of storage, but I will be able to use any wireless carrier, use Bluetooth headphones, and write an application that does whatever I want it to, without anyone stopping me.

On Management

I was fumbling through my business card holder just now, and discovered again a card I really appreciated when I got it. Some guy was installing a new Cisco firewall in one of my offices last year and I questioned him about his work. He gave me the usual, watered-down "it's a job" kind of thing, and gave me his card should I wish to seek employment alongside him.

This guy, Jeff (not important), worked for a company called Coleman Technologies, Inc. and it was apparently owned by Jeff Coleman (not the fellow on the front of the card). On the back was a pretty straightforward approach to leading. It was a list of 30 little statements, called Jeff Coleman's Laws. They are as follows:

1. No one is smart enough to be a dictator.
2. The only real power one has is the power of persuasion.
3. The less you know about something, the simpler it seems.
4. Important decisions require at least one night's sleep.
5. Decisions made without all the facts are guesses.
6. The most important thing a manager does is people picking.
7. Lies are hard to remember.
8. There is nothing more critical to true success than openness, honesty, and integrity.
9. Those that don't solicit and listen to advice are destined to be unsuccessful.
10. What is given cannot be taken away.
11. Meddling after responsibility is delegated and accepted, provides a built-in excuse for failure.
12. Unwritten agreements are soon forgotten.
13. Time is not a good decision-maker.
14. You must look successful to be successful.
15. Cash flow is more important than profit.
16. Grow or die.
17. The only people not making mistakes are those not doing anything.
18. Don't bite off more than you can bite off.
19. The most important and most difficult trait to identify is the ability to get things done.
20. A manager with a full calendar every day isn't delegating properly.
21. A full day spent in meetings is 40% wasted.
22. A pat on the back is the ultimate in cost effectiveness.
23. A manager that takes the credit for the work of the troops should be made a member of the troops.
24. A manager unwilling to take risks is destined for mediocrity.
25. Twenty percent of the people do eighty percent of the work.
26. People that feel comfortable in their job are more productive.
27. All contracts end.
28. The prepared bird gets the worm.
29. An unfilled position is better than one filled by the wrong person.
30. The killer of the bearer of bad news quickly joins the ranks of the uninformed.

After two minutes of clicking, I discovered that this is available at the Coleman Technologies website.

I realize that the phrases in this list are not original, but this is a great collection. I am now all out of motivation. Goodnight.

The iPhone Didn't Do It for Me

I'm surprised at Apple. Usually when something really groundbreaking comes out, the price of the item I've had my eye on drops into my range. In my case, it's been the 32GB iPod Touch for several months. I was glad to hear about the iPhone 3G, and even happier that the price of that was $199.* Almost two weeks later, the 32GB iPod is still $499. Why?

Is it that the price of the new iPhone is so low that Apple can't reduce the price of the iPod? Are there so many people dropping $500 bills on the biggest Touch that they don't worry about lowering that number?

Or is it just that the iPhone 3G is not that groundbreaking? I have a few reasons that might be the case:

1. Hundreds of thousands of people already have the pokey iPhone, and it's good enough for e-mail, so why change?
2. Solutions were built to make the first iPhone work for their business, so there really is no need to update to a new device that supports MS Exchange natively. It would make the recent infrastructure change a big waste of money.
   
3. They are late adopters, and only six months into the first two-year contract on the old iPhone. They simply can't afford to upgrade to a new device.
4. It wasn't impressive enough the first time to waste money on it again.
5. We're all waiting for Android on Open Handsets.
   

In any case, I'm still not buying the Touch until the price goes down. It will only take time, or the release of a 64GB or 128GB Touch. Fine with me. As soon as it hits $300 for the 32GB, I'm in. But I'm definitely not buying an iPhone.

*Price is $199 with a 2-year contract with AT&T wireless, the very worst wireless carrier in terms of customer service. Good luck if you think the $199 tag is worth it.

Firefox 3 Download Day

Even though I have two irregular readers, I'll tell them about this. Today is Firefox 3 Download Day, but the site doesn't have a link to download version 3 right now. It's 9:30 on the East Coast of the United States, so I would imagine the day has begun for most of the world.
Check back soon today and download Firefox 3. I'll be checking throughout the day and will update this post when I see they have provided a good link. Seems like they're wasting time today if they want to set a record.

Update: Looks like the downloading starts at 10AM PDT (that's California time), so my friend in Arizona (not McCain) can download starting at 11AM his time. 12 noon for Scott, and 1300 hours for me. The download day will then run for 24 hours, so you people have plenty of time to get it to all of your computers.

Update 2: It's 6:50AM on 6/18/2008, and download day is still going. Five more hours!

One Tell-Tale Sign You Might Be Getting Older

I went to my favorite Otolaryngologist today because I was feeling down and I couldn't breathe through my nose. He ended up prescribing so many pills that I had to buy this thing.

This, plus two nasal sprays.
Not much else to report today. I am, however, already feeling better.

The Glider

I guess I'm not as big a geek as I thought. I've never heard of the game of Life they speak of, but now I will explore it. Well, later. Right now I want to talk about a great article I read last night on How to Become a Hacker. I never thought I'd come across something like that, for real hacking is just that - hacking. Whatever art or science it is, if you're really good at it and are able to build things and solve problems, you are a hacker. A passion for such things is usually apparent in the person, too.

On to my subject, and a link. The first result from Google for 'amortization schedule' for years has always returned this one at FSU. I took a look at the fellow who wrote it and found this symbol on his page. It turned out to be a link and in that link were the words "hacker emblem." I had to look.

Apparently there is a following of people who use this emblem to mark themselves in this way, but according to the emblem guide it doesn't mark them as hackers; it only shows that they support the hacker culture. Fine with me. I'd rather not boast that I am a hacker, because if I'm the only one calling Ben Rehberg a hacker I obviously haven't proven to anyone else that I have the skills.

That said, I do support passionate homemade engineering, as I like to call it, so I will display the emblem on this site too. Come to think of it, I could put it on all the sites I own.

Not Dead Yet

The Web Crawler project is not dead. I'm fumbling computer hardware and software right now and making my best attempt at keeping up with school. Oh, and I have a job.

I have started hosting the project on Google's servers so that other programmers may join me in this. I haven't allowed access by anyone yet as I may not want help. The code, however, is available there (I think). Try looking at the rehbergwebindexer project if you wish. I don't think there is even any code in the source file yet; just commenting.

Consolidation

I'm sitting here this evening with a MacBook that has slowed to a crawl. It further delays my inevitable completion of a school project (fine with me), but I really just want to finish what I'm doing (not the school project) and go to bed.

It's not the MacBook. It's the parasite I installed on it.

For the past month, I've been juggling the Vista desktop I use at home for development, the Vista notebook I use for school, the Windows XP tablet that I have for my day job, and this wonderful MacBook that doesn't have many applications I actually use to produce things. It's cool to blog with it, chat, and play with the camera, but it's really just eye-candy. I can't do schoolwork with it (they require Office 2K7 documents), I can't find an FTP program for it, and I really can't figure out how to edit raw text - a very important feature I need to edit HTML and do programming.

To combat my two-computer dining room table, I installed Windows (the aforementioned parasite) on the MacBook using VMWare's VMFusion. It's a wonderful piece of software and it is very similar to Parallels, only cheaper ($40 vs. $80). I installed the trial of VMFusion, and an old copy of Windows XP. It has effectively slowed my MacBook to where it takes a full eight seconds to open a new tab in Firefox. It's working really hard right now on installing SP3, and I'm sure a slew of updates are in store after that is finished. It has Office 2007 and I shouldn't need much more to do everything that I need to do on this beautiful 13-inch MacBook.

The cool thing is that if I ever get really sick of the reduced speed, I can close the Virtual Machine and Windows goes away like a little troll in the closet. I feel powerful.

Okay, I know it's slow because I am only running 1GB RAM on this computer with two operating systems running. A fix (4GB) is on the way. After the updates and the memory upgrade I should have no problem. I might even install Ubuntu on another VM.

Must go now; I have to write a post to tell everyone that the Web Spider project is not dead - I'm just busy.

School of Ben

I'm attending school online, and while that doesn't seem very prestigious, it did present me with an idea this morning which I probably will not be able to digest very completely here this morning.

I can teach. And I can learn at the same time. I will post questions here for my readers to solve, and when they answer in the comments we can have a discussion, m'kay?

I started this post in the morning, and now it's late at night. I have forgotten the question I was going to ask. At least now you know my intent and you will know what your mission is when I choose to send you on one.

Schadenfreude

Like when George Bush fell off the Segway.

Break Time

I've sworn off the web crawler project for the week. I'm in North Georgia until Friday; I intend to ride my motorcycle and relax at the Bed & Breakfast I have enjoyed so far. I didn't bring any books related to the project (the bike was packed already) and I have to get some schoolwork done.

I might be hanging out listening to the rain tonight, but that's fine with me. As long as I can get home Friday, there won't be a problem. I want to ride some in the mountains while I'm here, but a friend is taking me to Athens tonight if the weather permits and it will probably rain Thursday. Friday I'll go home, likely without really hitting the curves up here. It was a nice ride up Monday; really good practice for the trip in September.

Back to "work" now.

Some Books

I got my courage up Saturday and ordered the books from O'Reilly. This press has long been highly regarded by technologists, whether they are programmers, IT professionals, or just geeks. Go ahead - ask a geek if he/she has a camel book, and chances are they'll know what you're talking about (and it will be within reach). Don't tell them what it is if they don't know.

I'm posting this to chronicle my efforts to build a web crawler and eventually a search engine. I expect to make further posts about how this project develops, and perhaps what I've found in these books that helped.

I have ordered three books. I went there for one, but there's always a deal to get three for the price of two, plus free shipping. And I can always find another book to get. So:

Perl & LWP. This one I've borrowed before, and it opened my eyes to the possibilities of automated web surfing using Perl. I built a small script one time that looked up my SMTP server's IP at spamcop, then e-mailed me if my mail server was ever blacklisted. It was fun and quite easy, but since I can't find that script right now I'll have to post it later.

Spidering Hacks. I ordered this one for obvious reasons. This book's excerpts is where I found that little bit on needing my spider registered. I expect to learn a lot and become very frustrated with what I find here.

Perl Cookbook. This was the third choice because I needed three. Also because it's $50 and I could use the discount. There apparently is a series of "cookbooks" that have really cool stuff (recipes) in them. There is also the PHP Cookbook, the C# 3.0 Cookbook, and more. I expect to find shortcuts and things I'd never thought of in this book.

Light Reading

I'm taking a class right now on software requirements engineering (does one actually engineer the requirements, or did they just want to make this class sound hard?) and I came across something I might use with the web crawler project.

In the chapter about "The Software Process" which talks about the processes necessary for an individual or team to succeed at building a quality piece of software or system, I came across the Personal Software Process, or PSP. The book simply states that every developer has a process, whether anyone can see it or not. Either way, there is a proper way to go about producing software at a personal level, and here is the gist (Pressman, 2005, p.37):

Planning. This activity isolates requirements and, based on these, develops both size and resource estimates. In addition, a defect estimate (the number of defects projected for the work) is made. All metrics are recorded on worksheets or templates. Finally, development tasks are identified and a project schedule is created.
High-level design. External specifications for each component to be constructed are developed and a component design is created. Prototypes are build when uncertainty exists. All issures are recorded and tracked.
High-level design review. Formal verification methods... are applied to uncover errors in the design. Metrics are maintained for all important tasks and work results.
Development. The component level design is refined and reviewed. Code is generated, reviewed, compiled, and tested. Metrics are maintained for all important tasks and work results.
Postmortem. Using the measures and metrics collected (a substantial amount of data that shoul be analyzed statistically), the effectiveness of the process is determined. Measures and metrics should provide guidance for modifying the process to improve its effectiveness.

I'm not sure if what I'm doing will fit into this personal model of development, but it's thought provoking. Even if I don't collect data about what my problems might be and then analyze the data about what actually went wrong, I can still hold myself to some kind of process. Even though I don't have a deadline or an antsy customer to deliver this to, I can possibly eliminate shortfalls if I just think it out before delving into code.

But then what fun would that be?


Reference (in our favorite APA format):

Pressman, R.S. (2005). Software engineering: A practitioner's approach. New York: McGraw-Hill.

Executive Decision

After toying with C# today, I've decided that it is way to process-intensive to write the application on a runtime environment like .NET or Java. What I need is a simple language that can download a page, rip through text like a bandit, write the necessary fields to the database, and move on. I can organize the data when the search engine extracts that data.

I can't commit to anything yet, but my spidey-sense is telling me that the crawler will be written in Perl with LWP. I suppose I could look at Ruby, too, but I already have my Camel book and have worked with LWP before. I haven't tied Perl to a RDBMS, but I have done it with PHP and it must be similar. Perl can also do some limited recursion from what I understand, and if it can't I may can use a database back-end to save the stacks of URLs.

I was ready to buy books at O'Reilly today (I chickened out of spending the money) and found a book on writing spiders. From the preview I surmised my crawler/spider must be registered. That means I have to go mainstream, doesn't it?

And now after some more reading, I have discovered that this crawler can be used to build an index for special purposes. I can build my own search engine for this site, for example, and get much better results than I can searching the Google index for benrehberg.com. I have searched for things I know I wrote about, but never found them with Google. Building my own search engine and maintaining my own index of the site can prove useful if I keep writing about programming.

Update: I have created a new label "Web Crawler" for all posts related to this project.

How to Write a Search Engine

It seems a bit strange using the world's best search engine to find out how to build your own. Google is my first resource in this project, though Google itself provides nothing but the idea. There is a paper at Stanford by Larry and Sergey, and that basically is the starting point. That is Google's only contribution so far aside from the many searches I will perform.

There are three main parts to the search engine: the crawler, which tirelessly captures data from the web, the database to hold everything, and the actual search engine - the queries that put the data together in a meaningful format for you.

I could write a search engine that actually crawls the web looking for my search criteria, but that is very VERY inefficient. Google (and many others) have solved this inefficiency by effectively downloading the Web (that's right - as much of it as they can) to their computers so it can search it much faster and have it available in one place. They've done a whole lot more to increase efficiency and effectiveness of searches, but downloading the web was the first thing they did. It turns out they needed a lot of computers.

I'm going to start with two. I have three desktops that no one wants to buy, and I am really tired of looking at them. I will probably need more if I get this index working soon, but there will be software considerations to make too. You can't fit the web on one computer, no matter how big. I will learn a lot.

I have always had an interest in distributed systems and cluster computing, so this will be fun. I have a lot to learn about distributed databases and algorithm analysis. But all that is later - I haven't even really finished thinking out the preliminaries yet. So one development/crawling machine, and one database machine. After I figure out how to crawl the web, I will begin work on performing searches. If this project holds my interest long enough, I might publish statistics at 49times.com, so keep looking. I will be posting here if I come up with anything worth publishing. I'm going to try to journal my progress and decisions without publishing code, but I realize that I very well could lose interest in this. If I get started, I will likely enjoy it and keep going, but no one can say. If you have some confidence that I will continue, you can subscribe to this blog and get the updates. Beware, though, that you'll get everything else I write too.

As a Student of Software Engineering,

from the stories I hear about glitches and compatibility and poor project management, this is friggin' scary.

Good Times

I realize we probably looked like a couple of homos walking down the beach, but my reunion with Scott was great. We drank, but not enough, and we didn't get tattoos either.

Just more reasons to do RAGBRAI together in 2010.

Uh-Oh.

49times.com is down since yesterday. You know it's on that powerful box, and I think it took the grid down for a few seconds yesterday when traffic was high. I can't tell yet, but I think there might have been as many as 2 simultaneous users, overloading the system.

Actually, the power blinked and I'm not home to restart it. Friday night is the soonest; I know you guys can't wait, but we all need to suffer a little bit for the cause.