Auto-Magical

Great image optimization tools for Windows and Mac

Over the last few weeks, I’ve experimented with image optimization tools. Using these tools, I have rapidly eliminated gigabytes of image data from thousands of images without any quality loss. Over time, this should translate to many terabytes of bandwidth savings.

Because these tools can be run in batch mode on thousands of images at a time, they are useful for optimizing large, existing image libraries. They are lossless and designed for bulk mode, which means you can safely run them without any loss in image quality. But be careful: test on small samples first and learn their specialties and quirks.

An alternative to running them locally is to use Yahoo! Smush.it, which is an online service that “uses optimization techniques specific to image format to [losslessly] remove unnecessary bytes from image files” using the same tools. The best way to run Smush.it is via the Yahoo! YSlow for Firebug, an add-on for the Firebug add-on for Firefox. (By the way, Smush.it renames GIF images to .gif.png when it shrinks them. I wrote a console app to rename them back to .gif. Browsers only look at the image header to identify images, so it’s safe to serve up PNG images with a .gif extension.)

Mac:

For OS X, all you need is ImageOptim, which optimizes JPEG, PNG, and GIF auto-magically. Seriously awesome tool. (Free.)

For lossy optimization, JPEGmini is amazing. It uses knowledge of human visual perception to shrink JPEG’s up to 5X without visible quality loss. (Semi-free.)

Windows:

RIOT (Radical Image Optimization Tool):

Though it has a batch mode, this is the best tool for optimizing single images, whether they are JPG, PNG, or GIF. I use RIOT to save every image I work on as well as to reduce the size of existing images that are too large. You can re-compress PNG and GIF images losslessly, but for JPG you want to save from the original file.

RIOT is available as a standalone version as well as a plugin for several image editors such as the excellent IrfanView.

RIOT - Radical Image Optimization Tool screenshot — RIOT – Radical Image Optimization Tool screenshot

The JPEG Reducer:

Run this tool in bulk on all your JPEG images to save ~10% of the file size. This is a GUI front end for jpegtran, which optimizes JPEG images by doing removing metadata and other non-display data from JPEG images. Because it is lossless, it is safe to run on all your image. It will ignore any files you add which are not really JPEG images.

PNG Gauntlet:

This tool is a front end for PNGOUT which will losslessly reduce the size of PNG images. Warning: if you add GIF or JPEG images to it, it will create PNG versions of those images. Sometimes you want to do this, but if not, don’t add images to the queue.

Let me know if you have other tools or ideas for image optimization.

JQuery VSDoc Url’s

I keep having to search for these URLs, so here’s a Note To Self:

VSDoc files are a feature of Visual Studio introduced for VS 2008 that provides IntelliSense for JavaScript. You can include it in your project, or you can reference the latest CDN copy of whatever library you use.

Reference it in your .js file like this:

///

And in your HTML like this:

<!--mce:0-->

In-memory cashing for an auto-complete list

When implementing an auto-complete control on your site, you may want to cache the results to keep the database queries to a minimum. Here’s a quick way to do that:

        public static Dictionary AutoCompleteHashList = new Dictionary();
        const string resultsFormatString ="{0}|{1}rn";
 
        private static string GetSuggestedResults(string s)
        {
            const int maxHashLengthToCache = 9;
 
            if (s.Length 
                            results.AppendFormat(resultsFormatString, recipe.Key, recipe.Value)
                );
 
            if (s.Length &lt; maxHashLengthToCache)
                AutoCompleteHashList.Add(s, results.ToString());
 
            return results.ToString();
        }

Stanza Catalog Update

I’ve continued enhancing the Stanza catalog I created earlier. New features include purchase links and Google Analytics tracking. I reverse-engineered the FeedBooks API and the AllRomanceBooks.com feeds to figure out how to do some of these things. Also I’ve been reading the OPDS spec (an industry-standard successor to the Stanza format) to add support for e-readers on other platforms. For example Aldiko officially supports OPDS, but it recognizes Stanza tags as well. It crashes if you try to open a PDF file however – so it just needs to be sent the right mime type.

Here’s the snippet where I reference the custom Analytics class I added:

 p.StoreDescription += Mises.Domain.Mobile.Analytics.GetAnalyticsImageTag(this.request.RequestContext.HttpContext);
                            var content = new TextSyndicationContent(p.StoreDescription,
                                                                     TextSyndicationContentKind.Html);
                            item.Content = content;

And here’s how to add external links to a book info view:

 item.Links.Add(
                                new SyndicationLink(
                                    new Uri(string.Format("http://mises.org/store/Product.aspx?ProductId={0}&amp;utm_source=MisesCatalog", p.ProductId)), "alternate", "Purchase at the Mises Store",
                                    "text/html", 0));

10 simple questions to evaluate a software development organization

All credit and inspiration for the questions below goes to Joel Spolsky’s The Joel Test and Bill Tudor’s 2010 update.

My version differs in two aspects: First, has just ten questions ordered from highest to lowest priority (in my personal opinion). Second, in addition to yes/no questions, it asks open-ended questions intended to give more informative answers in an interview or an analysis of a software team.

Do you use source control? What kind? What are the requirements/checks to check in code (assignment to work items, unit tests, peer review, etc)?
Do you use a bug database to track all issues? How do you track progress and manage change?
Do you use the best tools money can buy? For example: MSDN/Apple Dev accounts, dual monitors, and powerful workstations.
Do you have a dedicated QA team? Are they involved in the requirement/release management process?
Do you fix bugs and write new code at the same time? How do you balance the two?
Do programmers have quiet working conditions and team meeting rooms? Describe them.
Do you solicit feedback from end users or customers during the development process? How is it used?
Do you do a daily build? Do your builds include automated unit tests?
Do you have a requirements management system? Is it integrated with your source control?
Do you create specification/requirements documents? Do you do it before, during, or after writing code?

Bad developer ≠ novice developer

My post on the nature of programming seems to have struck a nerve. Many commenters pondered what makes a developer great. “Ka” thought that:

“You [are] not born [a] good or great programmer, you become one with time and study and hard work. At the beginning, everybody is a bad programmer.”

I disagree. Developers are not born “great,” but greatness does not automatically come with experience. Conversely, lack of experience does not make a developer “bad.” The difference between a great developer and a bad developer is not in their domain knowledge, but their methodology. The distinguishing mark of a great developer is that he codes consciously. Put another way, a good developer always knows why he is doing something. From the perspective of personal ethics, this requires intellectual courage and integrity.

Let me give an illustration of what I mean from personal experience:

When I got into Objective-C development, I was a “bad” developer. Most of my experience is with .Net code. Jumping into the iPhone dev world was intimidating. As as a result, I lacked the courage to learn the architecture. I tried to manipulate blocks of code found on the web without understanding what they were doing. I would copy and paste blocks of code and just change variable names. When things didn’t work, I would look for another block of code to substitute for the failing one, or enter “debugging hell” – running code over and over, making random changes and seeing if they worked. This is the hallmark of a bad developer – imitating without understanding. I kept this up for over a year. It’s not that I didn’t try to learn the language. I got several books and watched iTunes U classes. But the way I used the learning materials was to memorize blocks of code and look for places to stuff them into my code. I wasn’t actually learning the platform, just collecting samples. Some developers spend their entire careers this way. They carry collections of old code everywhere they go, and just grab chunks to insert into new programs. They may never select File => New File or File => New Project in their whole career.

After writing a lot of bad, buggy code, I drifted back to the comfort of .Net. Recently however, I decided to change my attitude. I started by downloading some iPhone code samples and open-source applications. I started in main.m and went through each line of code. If I didn’t understand exactly why a certain symbol was used or what it did, I looked it up. I spent a lot of time on Cocoa Dev Central, Developer.Apple.com, and Stack Overflow looking up things like the reasons why you would assign, retain, or copy a property, or when exactly you need to release an object (for alloc, new, copy or retain) or what you can do with respondsToSelector. There’s really not that much complexity to programming languages – but if you don’t take the time to learn how things work, they will always seem difficult and mysterious. If I had just looked this stuff up to begin with, I would have been far more productive. But, I was intimidated by the environment and tried to shortcut the learning process by imitating without understanding.

Understanding anything complex requires the courage and integrity to engage in difficult, exhausting mental effort. It’s tempting to cheat yourself. It’s easier spend more time in endless copying and debugging that take the effort to understand and create. In the short run, it saves time. But in the long run, developers who understand their craft are magnitudes more productive than the monkey see-monkey do coders. This is the difference between the unprincipled kind of laziness that trades understanding for time and the principled kind of laziness which saves time by understanding.

There’s no happy ending to my story — yet. The proof of a developer is in his work, not his book smarts, and I have yet to produce something to brag about.

For more on the traits of great developers, read these posts by Dave Child and micahel.

Get the MD5 hash of a file in .Net

MSDN has a page on how to get the MD5 hash of a string.

Easy enough, but if you follow their example exactly for a file and use File.ReadAllText() to get the string, you will get the wrong MD5 string for binary files. Instead, use File.ReadAllBytes() to bypass encoding issues. (This also applies to SHA1 hashing.)

private static string GetMD5Hash(string filePath)
        {
            byte[] computedHash = new MD5CryptoServiceProvider().ComputeHash(File.ReadAllBytes(filePath));
            var sBuilder = new StringBuilder();
            foreach (byte b in computedHash)
            {
                sBuilder.Append(b.ToString("x2").ToLower());
            }
            return sBuilder.ToString();
        }

Creating a Stanza Catalog with ASP.Net MVC 2.0

Stanza is a book reader for the iPhone/iPad. One of Stanza’s features is the ability to browse specially formatted book catalogs. While it has a number of built-in catalogs, you can also add your own. I have created such a catalog with ASP.Net MVC 2.0 (screenshots). The Stanza catalog format is pretty simple – just AtomPub with some proprietary attributes for images and things like search. This was a quick and easy project because the .Net Framework 4.0 has the System.ServiceModel.Syndication namespace which does all the RSS/Atom feed generation. We just have to add some custom attributes and serialize the feed to the browser.

Here is a quick overview of the code (Links are to the latest version of the source code in my SVN browser. You can get the project from SVN here (guest/guest).) The LiteratureCatalog and LiteratureCatalog.Tests projects have the relevant code.

Update: The Stanza catalog format works equally well with Aldiko, an e-reader for Android.

CatalogController.cs:

This is the default controller specified in global.asax. It defers to MisesFeeds to generate the feed items and to FeedResult to serialize and write out the feed.

Sample Method:

public FeedResult Journal(int journalId)
{
var feeds = new MisesFeeds(Request);
SyndicationFeed feed = feeds.GetJournalFeed(journalId);
 
return new FeedResult(new Atom10FeedFormatter(feed));
}

MisesFeeds.cs

MisesFeed contains all the code to generate a SyndicationFeed object containing a List of SyndicationItem. Note the Stanza-specific links added in search list-builder and the final helper method:

item.Links.Add(new SyndicationLink(new Uri(DataFormat.GetAbsoluteURL(p.Logo)),
"x-stanza-cover-image-thumbnail", "", "image/jpeg", 0));

public SyndicationFeed CreateFeedFromSyndicationItemList(IEnumerable postItems, string title,
string description)
{
var feed = new SyndicationFeed(title, description, new Uri(feedUri), postItems)
{
Copyright = new TextSyndicationContent(Configuration.Copyright),
Language = "en-US"
};
 
var self = new SyndicationLink(new Uri(Host + HttpUtility.UrlEncode("/Catalog/")), "self", "", Type, 0);
feed.Links.Add(self);
 
feed.Links.Add(new SyndicationLink(new Uri(Host + "/Catalog/Search/?q={searchTerms}",true),"search","Search Catalog",Type,0));
 
return feed;
}

FeedResult.cs:

FeedWriter inherits from ActionResult. It just writes the SyndicationFeed out with an XmlTextWriter:

public override void ExecuteResult(ControllerContext context)
{
if (context == null)
throw new ArgumentNullException("context");
 
HttpResponseBase response = context.HttpContext.Response;
 
response.ContentType = !string.IsNullOrEmpty(ContentType) ? ContentType : "application/atom+xml";
 
if (ContentEncoding != null)
response.ContentEncoding = ContentEncoding;
 
if (feed != null)
using (var xmlwriter = new XmlTextWriter(response.Output))
{
xmlwriter.Formatting = Formatting.Indented;
feed.WriteTo(xmlwriter);
}
}

Thanks to DamienG for the FeedResult class.

To see the catalog, get the Stanza app, tap “Get Books”, “Shared”, “Add Book Source”, then add the URL mises.org/catalog.

Some lesser-known truths about programming

My experience as a programmer has taught me a few things about writing software. Here are some things that people might find surprising about writing code:

Averaging over the lifetime of the project, a programmer spends about 10-20% of his time writing code, and most programmers write about 10-12 lines of code per day that goes into the final product, regardless of their skill level. Good programmers spend much of the other 90% thinking, researching, and experimenting to find the best design. Bad programmers spend much of that 90% debugging code by randomly making changes and seeing if they work.
A good programmer is ten times more productive than an average programmer. A great programmer is 20-100 times more productive than the average. This is not an exaggeration – studies since the 1960’s have consistently shown this. A bad programmer is not just unproductive – he will not only not get any work done, but create a lot of work and headaches for others to fix.
“A great lathe operator commands several times the wage of an average lathe operator, but a great writer of software code is worth 10,000 times the price of an average software writer.” –Bill Gates
Great programmers spend little of their time writing code – at least code that ends up in the final product. Programmers who spend much of their time writing code are too lazy, too ignorant, or too arrogant to find existing solutions to old problems. Great programmers are masters at recognizing and reusing common patterns. Good programmers are not afraid to refactor (rewrite) their code to reach the ideal design. Bad programmers write code which lacks conceptual integrity, non-redundancy, hierarchy, and patterns, and so is very difficult to refactor. It’s easier to throw away bad code and start over than to change it.
Software development obeys the laws of entropy, like any other process. Continuous change leads to software rot, which erodes the conceptual integrity of the original design. Software rot is unavoidable, but programmers who fail to take conceptual integrity into consideration create software that rots so so fast that it becomes worthless before it is even completed. Entropic failure of conceptual integrity is probably the most common reason for software project failure. (The second most common reason is delivering something other than what the customer wanted.) Software rot slows down progress exponentially, so many projects face exploding timelines and budgets before they are mercifully killed.
A 2004 study found that most software projects (51%) will fail in a critical aspect, and 15% will fail totally. This is an improvement since 1994, when 31% failed.
Although most software is made by teams, it is not a democratic activity. Usually, just one person is responsible for the design, and the rest of the team fills in the details.
Programming is hard work. It’s an intense mental activity. Good programmers think about their work 24/7. They write their most important code in the shower and in their dreams. Because the most important work is done away from a keyboard, software projects cannot be accelerated by spending more time in the office or adding more people to a project.

Add a Facebook Like button to your page

Adding a “Like” button using the Facebook JavaScript API is easy:

<div>
        &lt;fb:like href=&quot;"&gt;
    </div>
 
        window.fbAsyncInit = function () {
            FB.init({ appId: 'your app id', status: true, cookie: true,
                xfbml: true
            });
        };
        (function () {
            var e = document.createElement('script'); e.async = true;
            e.src = document.location.protocol +
      '//connect.facebook.net/en_US/all.js';
            document.getElementById('fb-root').appendChild(e);
        } ());