Schadenfreude

Like when George Bush fell off the Segway.

Break Time

I've sworn off the web crawler project for the week. I'm in North Georgia until Friday; I intend to ride my motorcycle and relax at the Bed & Breakfast I have enjoyed so far. I didn't bring any books related to the project (the bike was packed already) and I have to get some schoolwork done.

I might be hanging out listening to the rain tonight, but that's fine with me. As long as I can get home Friday, there won't be a problem. I want to ride some in the mountains while I'm here, but a friend is taking me to Athens tonight if the weather permits and it will probably rain Thursday. Friday I'll go home, likely without really hitting the curves up here. It was a nice ride up Monday; really good practice for the trip in September.

Back to "work" now.

Some Books

I got my courage up Saturday and ordered the books from O'Reilly. This press has long been highly regarded by technologists, whether they are programmers, IT professionals, or just geeks. Go ahead - ask a geek if he/she has a camel book, and chances are they'll know what you're talking about (and it will be within reach). Don't tell them what it is if they don't know.

I'm posting this to chronicle my efforts to build a web crawler and eventually a search engine. I expect to make further posts about how this project develops, and perhaps what I've found in these books that helped.

I have ordered three books. I went there for one, but there's always a deal to get three for the price of two, plus free shipping. And I can always find another book to get. So:

Perl & LWP. This one I've borrowed before, and it opened my eyes to the possibilities of automated web surfing using Perl. I built a small script one time that looked up my SMTP server's IP at spamcop, then e-mailed me if my mail server was ever blacklisted. It was fun and quite easy, but since I can't find that script right now I'll have to post it later.

Spidering Hacks. I ordered this one for obvious reasons. This book's excerpts is where I found that little bit on needing my spider registered. I expect to learn a lot and become very frustrated with what I find here.

Perl Cookbook. This was the third choice because I needed three. Also because it's $50 and I could use the discount. There apparently is a series of "cookbooks" that have really cool stuff (recipes) in them. There is also the PHP Cookbook, the C# 3.0 Cookbook, and more. I expect to find shortcuts and things I'd never thought of in this book.

Light Reading

I'm taking a class right now on software requirements engineering (does one actually engineer the requirements, or did they just want to make this class sound hard?) and I came across something I might use with the web crawler project.

In the chapter about "The Software Process" which talks about the processes necessary for an individual or team to succeed at building a quality piece of software or system, I came across the Personal Software Process, or PSP. The book simply states that every developer has a process, whether anyone can see it or not. Either way, there is a proper way to go about producing software at a personal level, and here is the gist (Pressman, 2005, p.37):

Planning. This activity isolates requirements and, based on these, develops both size and resource estimates. In addition, a defect estimate (the number of defects projected for the work) is made. All metrics are recorded on worksheets or templates. Finally, development tasks are identified and a project schedule is created.
High-level design. External specifications for each component to be constructed are developed and a component design is created. Prototypes are build when uncertainty exists. All issures are recorded and tracked.
High-level design review. Formal verification methods... are applied to uncover errors in the design. Metrics are maintained for all important tasks and work results.
Development. The component level design is refined and reviewed. Code is generated, reviewed, compiled, and tested. Metrics are maintained for all important tasks and work results.
Postmortem. Using the measures and metrics collected (a substantial amount of data that shoul be analyzed statistically), the effectiveness of the process is determined. Measures and metrics should provide guidance for modifying the process to improve its effectiveness.

I'm not sure if what I'm doing will fit into this personal model of development, but it's thought provoking. Even if I don't collect data about what my problems might be and then analyze the data about what actually went wrong, I can still hold myself to some kind of process. Even though I don't have a deadline or an antsy customer to deliver this to, I can possibly eliminate shortfalls if I just think it out before delving into code.

But then what fun would that be?


Reference (in our favorite APA format):

Pressman, R.S. (2005). Software engineering: A practitioner's approach. New York: McGraw-Hill.

Executive Decision

After toying with C# today, I've decided that it is way to process-intensive to write the application on a runtime environment like .NET or Java. What I need is a simple language that can download a page, rip through text like a bandit, write the necessary fields to the database, and move on. I can organize the data when the search engine extracts that data.

I can't commit to anything yet, but my spidey-sense is telling me that the crawler will be written in Perl with LWP. I suppose I could look at Ruby, too, but I already have my Camel book and have worked with LWP before. I haven't tied Perl to a RDBMS, but I have done it with PHP and it must be similar. Perl can also do some limited recursion from what I understand, and if it can't I may can use a database back-end to save the stacks of URLs.

I was ready to buy books at O'Reilly today (I chickened out of spending the money) and found a book on writing spiders. From the preview I surmised my crawler/spider must be registered. That means I have to go mainstream, doesn't it?

And now after some more reading, I have discovered that this crawler can be used to build an index for special purposes. I can build my own search engine for this site, for example, and get much better results than I can searching the Google index for benrehberg.com. I have searched for things I know I wrote about, but never found them with Google. Building my own search engine and maintaining my own index of the site can prove useful if I keep writing about programming.

Update: I have created a new label "Web Crawler" for all posts related to this project.

How to Write a Search Engine

It seems a bit strange using the world's best search engine to find out how to build your own. Google is my first resource in this project, though Google itself provides nothing but the idea. There is a paper at Stanford by Larry and Sergey, and that basically is the starting point. That is Google's only contribution so far aside from the many searches I will perform.

There are three main parts to the search engine: the crawler, which tirelessly captures data from the web, the database to hold everything, and the actual search engine - the queries that put the data together in a meaningful format for you.

I could write a search engine that actually crawls the web looking for my search criteria, but that is very VERY inefficient. Google (and many others) have solved this inefficiency by effectively downloading the Web (that's right - as much of it as they can) to their computers so it can search it much faster and have it available in one place. They've done a whole lot more to increase efficiency and effectiveness of searches, but downloading the web was the first thing they did. It turns out they needed a lot of computers.

I'm going to start with two. I have three desktops that no one wants to buy, and I am really tired of looking at them. I will probably need more if I get this index working soon, but there will be software considerations to make too. You can't fit the web on one computer, no matter how big. I will learn a lot.

I have always had an interest in distributed systems and cluster computing, so this will be fun. I have a lot to learn about distributed databases and algorithm analysis. But all that is later - I haven't even really finished thinking out the preliminaries yet. So one development/crawling machine, and one database machine. After I figure out how to crawl the web, I will begin work on performing searches. If this project holds my interest long enough, I might publish statistics at 49times.com, so keep looking. I will be posting here if I come up with anything worth publishing. I'm going to try to journal my progress and decisions without publishing code, but I realize that I very well could lose interest in this. If I get started, I will likely enjoy it and keep going, but no one can say. If you have some confidence that I will continue, you can subscribe to this blog and get the updates. Beware, though, that you'll get everything else I write too.

As a Student of Software Engineering,

from the stories I hear about glitches and compatibility and poor project management, this is friggin' scary.

Good Times

I realize we probably looked like a couple of homos walking down the beach, but my reunion with Scott was great. We drank, but not enough, and we didn't get tattoos either.

Just more reasons to do RAGBRAI together in 2010.

Uh-Oh.

49times.com is down since yesterday. You know it's on that powerful box, and I think it took the grid down for a few seconds yesterday when traffic was high. I can't tell yet, but I think there might have been as many as 2 simultaneous users, overloading the system.

Actually, the power blinked and I'm not home to restart it. Friday night is the soonest; I know you guys can't wait, but we all need to suffer a little bit for the cause.

And I'm Spent...

It is working. After a long battle all day yesterday (and giving up on Apache), Ruby on Rails is running. The rest of my configuration is yet to be done (no database yet), but all in good time. Take nothing for granted: this is a very powerful server. Here are the specs (and yes, it is 2008):

Fedora Core 8
450MHz Pentium II
512MB RAM
10GB HDD

Should serve very well for the amount of traffic I expect at 49times.com.

Should Have Been Mine

An article in the Washington Post today caught my eye. It's about a web-based startup called WEBook that lets authors collaborate on a novel and they all make money when it's published. Some of the books will be sold through Amazon's on-demand publishing and some sold directly at Barnes & Noble stores.

For some reason, this sounds really familiar. I don't know why, but it may be similar to an idea I had several months ago. They probably had the idea at that time, but they were smart and got a company going, with investors and everything.

Which leads me to believe that I in fact do have good ideas from time to time. No one batted an eye when I mentioned a community-written book, and now a few folks are living it up in Mountain View with their good idea. The model can last so long as there are serious players writing there. My brother could benefit from something like that.

Now, what to do with my next big product of random thought before someone makes a corporation out of it...

Never be Late Again

With Gmail's Custom Time, just make up an event in the past and say it happened. It's that easy!

You may even figure out a way to win last week's lottery using the Custom Time API! I'm going to create an app for Android so you can even keep a little slice of your own time in your pocket (coming the second half of 2008). But when that happens, I'll have had it since 2005.

You guys are way behind!

Gimme a Minute...

I used to get a locally roasted and flavored coffee called Highlander Grog downtown in Colorado Springs. It was always a special time when I was able to get it. I took a thermos of it to Literature class on Friday nights, sat up in the morning and drank it, and winter was the best season to have some Highlander Grog. Even more special were the times the coffee shop was actually brewing it, and my daily coffee turned into an aromatic treat.

And then I moved to South Georgia. You've heard enough about my regrets of this decision so I won't bore you with that. But down here, there are quite a few people who haven't even heard of the television show Highlander, or even muttered the word before. They think they know coffee when they get some bag at the grocery store that says "Gourmet" on it. They have no idea.

My parents took a road trip up the East Coast in late 2005 and discovered Intercourse Canning Company in Intercourse, Pennsylvania. Guess what flavored coffee they have!

So every once in awhile my mom orders coffee from there. This morning is the first time in over a year I have smelled that familiar aroma that takes me back in time. I stood at the coffee pot a few minutes ago with my eyes closed, and I could almost see my Literature instructor talking about Othello. I could feel campus around me. I was back to a happy time, long ago.

So to combat this, I'm sitting at my kitchen table in Adel, Georgia, with a cup of Highlander Grog under my nose, waiting for my daughter to wake up. I can enjoy my good memories from time to time, but that never means I have to stop making them.

Good Point.

Barack Obama doesn't like the VP idea. From CNN:

"'I don't understand,' he said. 'If I'm not ready, how is it that you think I should be such a great vice president?'"

I Think She Knows

This isn't me, by the way. It's John Mayer. If it won't play here, it might over there.

Bono:

"The less you know, the more you believe."

That got me going this morning. I love it!

Reflections

Just testing a cool script I found today:

Subsidies

"It's the opinion of some that crops could be grown on the moon; which raises the fear that it may not be long before we're paying somebody not to."
Franklin P. Jones

I don't know why this struck me as very funny this morning.

Friends with Vista

After nearly a year, I finally decided to figure out what I could do to make my Vista laptop a bit faster. The memory is maxed out at 2 Gigabytes and it has a dual-core AMD CPU. It had always been very very slow in completing trivial tasks, like opening a browser or the control panel. Copying and moving files took way too long, and I just never approached my problem with logic.

A few weeks ago I was talking with a friend about my experience with Vista so far, and mentioned to him that I didn't think it was a problem with Vista, but a hardware issue with my Gateway laptop. "It runs very hot," I told him. "The hard drive activity never stops. I just don't think the machine was designed well enough to support such a heavy OS." I'd never seen Vista so slow on any other computer, so why the hell is it pokey on mine? And what in the dickens is going on with my hard drive?

Then it hit me. Constant hard drive activity is an indicator of (1) a virus or crapware, or (2) an indexing service. Google Desktop search was deployed with the computer when I bought it; part of Gateway's image, along with all the other garbage like BigFix, AOL , and the Office 2007 90-day trial.

Having been a student of Vista before and during its release, I remembered something about Google and Microsoft having fits about desktop search. It seems that Vista includes its own indexing service to speed up searching, and Google was having a hissy over users not being able to choose a desktop search engine. The Windows Indexing Service is on by default, and I don't think any manufacturers have changed that in their production images. And it just so happens that Gateway included Google Desktop in every computer they released with Vista, and therein lies my problem: two indexing services, constantly running on my poor little 5400 RPM notebook hard drive.

After some thought, I decided I'm a fairly organized fellow and don't have the need very often to search for a document. Most of what I access anyway is on the network, and those locations aren't indexed by default anyway. So away went Google Desktop. Though I love Google, I have no need for that program on my mobile station.

And for that matter, I canceled the Windows Indexing service. No need to pick sides, you know?

Then for a final pick-me-up, I had Vista optimize the graphics for performance, which took away all the eye-candy and effectively made my desktop look like Windows 2000. I'm fine with that.

Oh, and one more thing: I shut off the UAC. Those pain-in-the-ass messages one gets when he tries to install a program, "Windows needs your permission to continue," are gone. I can now run a command window without specifying to run it as Administrator. I can change IP settings with fewer mouse clicks. A little bubble message when I log on warning me that User Account Control is turned off is the only annoyance I have now, and I'm sure that with a simple registry edit I can get rid of that too. Maybe I'll post it later.

I must say this little bottom-end laptop is pretty damn speedy these days. NetBeans opens in under 60 seconds. Outlook opens in under 5, and boot times are at their lowest since I got it. This doesn't change anything about the inevitable change to a Mac when I can afford one, but it certainly makes me more comfortable in delaying it.

HD Coasters

It's in all the latest news, so I might as well chime in on it, too. HD-DVD is dying, and fast. Since Warner Bros. announced that it is dropping it from its DVD release formats in January, Netflix said they're not going to place any more orders for movies in HD-DVD. Then today I read that Wal-Mart is phasing it out. Seems like there's a clear dominant format finally.

Geez, before I can even finish writing this, Toshiba calls it quits. Forget I said anything, then. I guess it's really over. At least I didn't get the Xbox 360 with the HD-DVD player. Now there's 300,000 pissed-off people if you ask me. Hell, it took me until 2001 to buy a DVD player. Sometime next year I'll have a Blu-ray device, but not now.