XCFLabs.com

The Power of One

In August of 1999 I took a leave-of-absence from Sun Microsystems, thinking of what I should do next. I was 21 years old and within a week I knew what I wanted to do. I was going to do a web-based app that would change the world. Right.

To do a web-based app one needs to write code that emits HTML in live response to a user's clicks. At the time, there were some proprietary solutions to help programmers do that and there were two standards: Microsoft's ASP (Active Server Pages) and Sun's JSP (Java Server Pages), supported by a handful of very expensive web servers.

I wanted to use the Apache web-server for my project, however. It was free, easy to set up on Linux, and ran more than 60% of the sites on the Web. I could write JSP for Apache using GNUJSP - a free implementation of Sun's standard that worked with Apache via JServ, a piece of free software that allowed Java Servlets to run with Apache.

GNUJSP was written by Vincent Partington, a programmer in the Netherlands. There were quite a few users and it was well-supported. Unfortunately, GNUJSP was written according to Sun's alpha spec of Java Server Pages. When the specification was finalized, it looked very different from the alpha version. The syntax had changed dramatically and many new features were added. Vincent announced that he would not work on a 1.0 implementation.

As is usually the case, Vincent's message signaled an opportunity to several open-source programmers. They would continue his work. When I first looked at GNUJSP, several had started 1.0 implementations and announced their efforts on the GNUJSP mailing list. This work was very much needed, because other Apache users - not just some impatient 21-year old - were anxious to use 1.0 features.

Being 21 has its advantages: you don't second-guess yourself. "I can do this in three days, max," I thought. And I did. I started from scratch and typed in the code that was brewing in my head. I remember staying up till 5am the first night, then working 3-4 hours on it during each of the next two days.

It helped that I had worked for 2 years at the Java Server Group in JavaSoft prior to this, but I never looked at Sun's reference implementation nor discussed JSP with Anselm Baird-Smith or Satish Dharmaraj - the two guys down the hall that conceived the Java Server Pages. I did have to solve some theoretically idiotic (or, rather, practically unavoidable) problems. For example, the Java Servlet API implementation by Sun that was publicly available at the time was version 2.0. The new features of JSP specification 1.0 mapped directly to new features in version 2.1 of the Servlet API, but at the time there was no freeware implementation of 2.1. I had to make do with what I had, writing some rather horrible hacks.

After 3 days of hacking I started using my code and moved the web application I was writing to use it. Most things appeared to work, so with the confidence of a management consultant I sent an email to the gnujsp mailing list. I described my work and where the code was located. Within 24 hours I had the first bug reports.

The open-source model of software development works because of the human necessity to show others how smart they are. Everyone wants to not only find a bug but also one-up the author and point out the fix. I and gnujsp benefited from this greatly. Over the next week I would receive as many as 5 bug-fixes per day. I integrated the fixes, monitored the mailing list for general problems, and posted new versions as often as twice per day.

After seven days the code stabilized. I checked it into the project's CVS tree, but left old 0.9 version there just in case. Other programmers were now able to work with my code. They continued to fix bugs and eliminate inconsistencies and gnujsp wouldn't be what it is today without them. The new version became known as GNUJSP 1.0, but, since I started it from scratch, from then on I always referred to it as "my gnujsp."

As the old users were migrating, potential users must have liked seeing that the software was so well-supported and the mailing list was now full of messages by people trying to get started. The project now had several people actively monitoring the list for problems encountered by the new users, so I decided to step away and unsubscribed from the gnujsp mailing list.

While I was working on gnujsp, I was also looking over the work in the surrounding landscape. I couldn't have been the only person quick enough to implement the JSP 1.0 spec and couldn't have been the only one to distribute it under some freeware license. I saw two other projects, both done by lone programmers. One had used an XML parser, which I knew was a rather indirect and clumsy approach for parsing JSP source files. The other project - PolyJSP - seemed to be fast and complete, but did not appear to have any users, even though it claimed to be retargetable to source languages other than Java - a very cool feature, but one that I knew pragmatic users did not care for.

The competing effort that did scare me was a publicized attempt by a Sun engineer to convince Sun management to release Sun's own JSP and Servlet implementations under the Apache license. The Apache license allows anyone to use and re-distribute code with the only restriction that the Apache name not be used to endorse or promote derived work. Essentially, free to do whatever.

The Sun engineer working on releasing the code was communicating directly with the Apache team and aiming to give Sun's implementation directly to the Apache project, so that the servlet engine and the jsp component would be tightly integrated with the Apache server. This would increase by several orders of mangitude the immediate deployment environment for Sun's code, which was previously restricted to the Java Web Server (a web server running only several hundred sites) and the Java Servlet Development Kit (a developer kit used by many programmers, but not used by any deployed sites.)

The Sun implementation was finally donated two months later amid much fanfare, with cool project code-names, promises of tighter integration with Apache, and things like a "Project Management Committees". Tomcat, the name given to the code, was difficult to set up, somewhat unreliable, and slow after two iterations of coding done by people other than Anselm and Satish. But it had the mind-share of the Apache group, the developers of the "Java Apache" project, and its "Project Management Committee." It was the status quo.

It took Tomcat another 3-4 months to become stable. During these six months many new users continued to choose gnujsp. A long article on the Sun developer site in the Spring of 2000 detailed how a company called Clip2 embraced the Java server architecture. Clip2's framework was rather complex. But they used gnujsp.

(Clip2 later exhausted its' $20+ million financing and changed its plan to R&D in peer-to-peer technologies. When I was doing InfraSearch in the Fall of 2000, we tried to buy Clip2. They later went out of business.)

As Tomcat was maturing, programmers began to take note. Most new sites chose it or a commercial implementation instead of gnujsp. Nobody on the gnujsp project was prepared to implement further revisions of the JSP specification, while the Sun team promised to ensure that the code would stay synchronized with the spec.

Gnujsp gave programmers in time of transition and uncertainty a free tool to build things of value to users: applications. Many did, but I considered it a rather humble success against the background of failure of the reason I programmed it: my own web site.

Did I fail with my own site because I had taken two weeks out of my development time? Perhaps that was not the only reason, but I could have certainly remained focused on my application for two precious weeks if I was only able to live with the shortcomings of off-the-shelf tools.

Today is January 18, 2002, two and a half years after I spent 3 days writing GNUJSP 1.0 and about two weeks integrating bug-fixes from first users. JSP implementations are now commodity products and every web server, application/e-commerce server, freeware or commerical, includes one.

The event that prompted me to write this is an email I received from someone who was facing the task of fixing a problem with a site that is still running GNUJSP. Occasionally I receive such email because people overlook the address of the mailing list and send messages directly to the listed authors. The day after I received the email I was talking to a friend who brought up my way of thinking/working, asking "Don't you like 'bringing the product to the people'?" "Yes, but hey, I've failed at that every time." After a short pause I remembered: "Wait, I just got an email from some guy who is still running gnujsp and is trying to solve some problem with it now!"

At home later that night I wanted to forward the email to my friend. Before hitting "Send" I thought I'd check if anyone was still running my code. Yes, my code. The instructions I typed in years ago.

I went to http://www.klomp.org/gnujsp, the site started by Vincent Partington when he started working on GNUJSP. It was still there, the last news item referring to an October 2000 release of version 1.0.1. How did it happen? Turns out Carsten Heyl continued integrating bug-fixes into the source-code.

What was I doing in October 2000? "Doesn't seem like that long ago," I thought. I was working on InfraSearch. The neurons of nostalgia fired up in my 24-year old brain. "Sigh, I'll search Google, maybe someone remembers."

People do remember. On the first search results page I saw a link to someone's resume mentioning GNUJSP in the paragraph listing the tools he's worked with, along with Java, C++, XML, and other technologies. Subsequent search-result pages listed 10 other resumes.

When 10 people were composing their resumes, they made a conscious decision that listing gnujsp would help them in their quest for a better job.

 Among the search results I also saw packaged binary distributions for RedHat Linux, Debian Linux, FreeBSD, and for Windows. Someone not only went out of their way to create them, but also had assumed that the task would be worthwhile - that others would download and use them.

Sites like vindigo.com, developer.novell.com, and many others still run gnujsp. Among the oddities I was able to locate within a matter of minutes were The USDA Forest Service Southern Research Station, the Canadian Film Centre's Story Engine, some random education site in England, some Astronomy site, and the portal for technically-minded Lutherans where you can put your very own Lutheran community online with 200MB of disk space, Apache, JServ, gnujsp, and mSQL. If you aren't convinced, check out a list of farms in your area.

"The USDA Forest Service Southern Research Station's mission is to create the science and technology needed to sustain and enhance southern forest ecosystems and the benefits they provide." Wow. Did I ever think this place would run the instructions I typed in? Of course not.

Noel Bergman, a significant contributor to the project after I left, mentions the project in his biographical blurb at IBM developerWorks. Hope you are charging big bucks for your consulting hours, Noel!

Many freeware web-servers that don't have their own JSP implementations still show instructions on how to use gnujsp with them. The World-Wide-Web Consortium's Jigsaw server does, and so does Jetty, a tiny web server popular in the embedded devices niche.

Turns out that someone posted instructions for installing gnujsp with Apache on a Cobalt RAQ3, on Mac OS X, and on the Lotus Domino Web-Server. There is even a Lotus Domino bug submission related to gnujsp.

I have never heard of vqServer, but someone had invested the time into writing a how to for it.

I was most pleased to see that someone had commented on how nicely gnujsp displays errors. This is one of the best things software developers can hope to hear from their users. From a post on another mailing list:

> Problem #5) Useless Error Messages
 > The error messages are generated by the JSP implementation
> and are completely independent of what JSP is. GNUJSP > had very helpful error messages and gave you the line number
> in the JSP page itself where the error occurred (not in the
> servlet like JRun does). So this is a complaint against
> the implementors of JSP, not JSP itself.

The code was also used in completely unintended ways when some internet appliance shipped with it and when someone had written an article on how to use gnujsp to generate WML.

Yes, this is a miniscule result when compared to achievements of authors of many pieces of software. The browser you are using to read this page, the server, the editor with which it was written, the operating system on both sides, the network routing layers, all have been written by individuals. When thinking of the multitude of software people use every day, gnujsp is off somewhere in a corner of the last row, a shy kid in a class picture weakly smiling at the camera. Yet it is result I am proud of. Each individual contribution to gnujsp increased the abilities of other programmers. It showed the power of one person's effort.

Making your own Yahoo! Messenger IMVironments

When Yahoo! Messenger installs, it registers a URL handler for ymsgr:// type urls with IE. When you click on a ymsgr:getimv?xxx link, it tries to retrieve http://messenger.yahoo.com/getimv/xxx.txt. The reason I had to patch the Messenger executable was so that it would look for these files at http://imvironments.org/dl/getimv.

Here's an example of what this txt file contains:
imv=smashbros
cab=http://us.dl1.yimg.com/download.yahoo.com/dl/imv/smashbros.yim
name=Super Smash Bros. Melee
vendor=nintendo
id=0JhnpBThTka1xNdae_Tv4sjJ9Sp1hGobEIk_BgkXWgl3dH4-
cmd=install

The id value is actually not base64, because some imvironments have ids that aren't padded correctly according to base64 rules. To generate your own, base64-encoding an MD5 checksum should be good-enough.

The cab value points to a Microsoft CABinet file, but with a .yim extention.

The cmd value can be install or remove, but remove doesn't seem to work reliably.

The YIM files can be unzipped (uncabbed?) with Microsoft's CABinet extraction tool.

Once you run it on a .yim file, you'll see some files in imvcache/xxx/, where xxx is the name of the imvironment. Here's an example listing:

Directory of C:srcyimsrcimvcachesnowflake

07/24/2001  05:36p3,298 clouds1.gif
10/21/2001  09:54a10,598 preview.gif
07/24/2001  05:36p497 snow2.gif
08/07/2001  03:03p5,674 snowman.gif
07/26/2001  03:42p3,734 splat.wav
07/24/2001  05:36p4,093 splat1.gif
11/04/2001  03:21p  995 sflake1.gif
11/04/2001  03:22p  943 sflake2.gif
11/04/2001  03:24p  899 sflake3.gif
11/04/2001  03:27p;913 sflake4.gif
09/27/2001  06:01p4,736 snowflake.imv

The .gif and .wav files should be self-explanatory. Hack away ;) I also saw that some environments contain Flash (.swf).

When you are done, you can create a cab file with Microsoft's makecab.exe. I used a GUI tool called CabPack. (Make sure you check the "Microsoft compression" box, rather than "LZW".)

When you are done, send it to us and we'll generate the corresponding .txt file and put both up for download. We are now working on an automatic way to do this. That page will be shown to the user when he/she clicks on the "Get more IMVironments" link within Yahoo! Messenger.

The big question, though, is what do the .imv files contain? If you look at snowlake.imv, it appears to be closest to base64. When I ran it through a base64 decoder, it seemed to produce a binary file, although the imv files of some other imvironments weren't padded according to base64 rules.

My intuition (and probably everyone else's) is that these files are html/javascript, although if Yahoo really does use base64 encoding, then maybe the binary output I saw was bytecode for some javascript VM? I hope that people explore this further, because until we can write our own, we can only modify and distribute hacks of the ones Yahoo makes.

 

-----Original Message-----

From: "Laura Covington"
Sent: Fri, 11 Jan 2002 10:51:30 -0800
Subject: imvironments.org;

Dear Abuse Team:

I am writing because it has recently come to our attention that through Maxim.net, you are hosting a site, http://www.imvironments.org, which is actively promoting the hacking of Yahoo! Messenger environments.  Their activities violate our valuable trademark and copyrights and also violate your own terms of service.  We would appreciate it if you would immediately deactivate this site and terminate the user's account.  

Please let me know if you need more information from us in order to proceed or if you have additional information regarding this site which you believe to be relevant to our concerns.  

Many thanks for your time. I know you are busy!

Best regards,
Laura Covington
Associate General Counsel Yahoo! Inc.
Telephone: 408.349.5187
Fax: 408.349.3400<
E-mail: lhc@yahoo-inc.com

XCFLabs.com