Prefinery

We use Prefinery at Prefinery, of course.

When you visit our signup page and request an invitation code, a Prefinery-powered form collects your name, email address, whether you are in the Web or Desktop software business, when you plan on launching your beta, and how you heard about us.

A daily e-mail digest of new testers, as well as an RSS feed, lets us know when we’ve got new signups.

When we got TechCrunched we gave out a special invitation code (TECHCRUNCH) good for 100 uses.  When TechCrunch readers went to our site to sign up, we validated the invitation code using the Prefinery API.

Using our own analytics we monitor graphs showing the rate at which new testers sign up.  We know our conversion rate (views of our signup form to actual signups) is 37%!  Our analytics also tell us that 75% of our testers are creating Web software and only 25% are working on Desktop applications.

Sending an e-mail survey to all beta testers was incredibly easy.  From within Prefinery we exported all testers to Campaign Monitor in a matter of seconds.

We are our biggest customer.  You should be yours.

Here are just some of the ways that being our biggest customer and eating our own dog food helped us make a better product:

1. We became experts in our domain.

We refined our own beta management process along the way - everything from collecting feedback to communicating with our customers.  We’re baking these lessons right into our product to help our customers run a more successful beta program.

By being your biggest customer you have no choice but to become an expert in your field.

2. We quickly discovered what must-have features were missing.

Once we started managing our own beta of a couple hundred testers we quickly became frustrated with our own product.

3.  We kept ourselves in line.

Not only did we discover new features, but in some cases we invalidated existing features.  We found ourselves asking, “Why did we build that?“  or “Do we really need this?“  Best of all, we never had to wonder “What do our customers want?”

4. We performed real end-to-end testing, continuously.

It’s easy to lose sight of the big picture.  You’re so busy testing individual features that you miss stepping through the product, just as your customer would.  By living in our app we caught bugs way before our customers.  Furthermore, we often found that the flow, or experience, of a given task just didn’t feel right once all the pieces were put together.  Best of all, we got this end-to-end testing for free.

Measuring the success of your beta program is unquestionably important and lots of powerful analytics are in the works over here at Prefinery.  Here’s our first of many graphs and charts to come - Tester Signups over Time.

We’ll show you the growth of new tester signups as well as the number of times your invitation form was viewed (we call these impressions).  These two metrics tell you how many visitors you’re converting into beta testers.

What other analytics would you like to see?

We’ve been working incredibly hard to improve the heart of Prefinery - creating your invitation form.  Without any further ado, we present to you …

Here are just some of the exciting improvements:

• Real-time preview
• Slick drag-and-drop interface
• Re-order all questions, not just the ‘additional questions’
• All questions can be marked required or optional
• Add custom HTML to your form; great for additional instructions or style enhancements
• New ‘Agreement’ question type can be required, as in “Do you agree to our Terms?” or optional, as in “Do you want to receive our newsletter?”
• Customize the Thank You, Error, and Submit button text
• Customize the labels for the contact information fields (name, email, address, etc.)

Hard to believe it used to look like this:

What do you think?

How do you get your beta testers to give you feedback?  Over the past 3 months I’ve run a series of experiments on Prefiney, including feedback links, polls, surveys, and email.

Add a Feedback Link

My first experiment was to add a UserVoice feedback link.  The link is prominently displayed on every page.  You can’t miss it.  When you click the link, a little popup box appears where you can quickly leave your opinion, whether it be a suggestion, feature request or bug report.  Leaving feedback couldn’t be easier.

Over the past 3 months a total of 2 of Prefinery’s approximately 400 beta testers have given feedback using the feedback link.  Over this same period, 5 people emailed our support address to give feedback.  The Contact Us email link is tiny and concealed at the bottom of the page.  What gives?

The hypothesis here is simple - an email goes to a person and there’s an expectation that someone will receive your suggestion and follow up with you.  People like to talk to people, so place your contact link, email address, or phone number prominently on your site. Better yet, use Campfire or Olark to let your customers chat with you instantly.

I believe that you’ll convince little more than 1% of your traffic to click your feedback link and leave a comment. Now, if you have a very popular site, 1% is nothing to scoff at.  But, if you’re just launching your web app in private beta, 1% might as well be 0.  Sure, you should still place the link on your site.  But, you’ll want calibrate your expectations.  Better yet, actively reach out to your testers by asking them to take a poll or survey …

Publish Inline Polls

In the sidebar of one of Prefinery’s most popular pages we have begun placing a small poll.  We ask simple questions, with a yes or no response.  Such as, “Want to see tester growth over time on a chart?” or “Want to be able to create polls like this one with Prefinery?“   The polls are powered by PollDaddy. 

Every week we swap out the question via our PollDaddy account, with no changes to Prefinery itself.  Over the past month we have run three polls and had a good response to each.  About 3% of testers take the polls.  Polls are quick and easy for both the publisher and the respondant and have certainly been a success.

Email a Survey

Using Survs, we created a Beta Satisfaction and Pricing Survey (you can re-use our survey template here) which asks typical questions, such as “What did you like (dislike)?”

We very much wanted to hear the bad news alongside the good.  It was imperative that testers who tried Prefinery and decided not to use it answer the survey and tell us how we could improve.  So, we decided to offer an incentive for taking the survey - a $50 gift certificate to Amazon or Apple, winner’s choice, to be awarded to one lucky participant.  Also, we only asked those who chose not to use Prefinery one question - “Why not?”

The survey was distributed via multiple channels - we emailed it to 414 beta testers, tweeted about it from @prefinery, and linked to it from within our user’s admin account.  Survs creates a unique link for each channel, so we could measure the success of each.  The actual email was sent using Campaign Monitor, which also has very impressive analytics and is perfect for sending one-off mass emails.

Of the 414 beta testers who received our survey, 155 opened the email.  A typical open rate for technology-related email is 20%, so we are very pleased with 37%.  A grand total of 59 (14%) of all testers completed the survey.  37 (9%) took the survey after receiving the email.  Of our 94 Twitter followers, 9 (10%) took the survey.  A shocking few 13 (3% of all testers) followed the link from within their user account!  The conclusion?  Asking your customers to take your survey actually works!

Summary

14% of all testers took the survey, 3% took the inline polls, and 0.5% submitted feedback using the feedback link.

Overall, the experiments showed that passively placing your contact information, a feedback link, or even a survey on your site is not enough.  If you want to know what your users think of your product, ask them.

I like the idea of building a business around owning a single link on a Web page.  Prefinery owns the Beta link.  UserVoice and GetSatisfaction share the Feedback link, while Tender and ZenDesk split the Support link.  Most recently, Inquiry is going after the FAQ link.

What’s still up for grabs?  Perhaps the About or Contact pages?  The legal links — Terms of Service and Privacy Policy — may be a stretch.  Certainly the API and Status links would be good niches.

Beta is a mindset — not a product life-cycle stage.

When in beta you’re focused on working with your customers to build a better product.  Why would you ever want to stop listening to your customers?  Why would you ever want to stop improving?

The term beta has been given a bad rap.  Beta’s good name has been soiled by companies’ overuse of the term — slapping the label carelessly on products for indefinite periods of time.  I couldn’t agree more with 37signals’ Getting Real, which encourages not using beta as a cop out for your lack of commitment to delivering a finished product.  That’s certainly bullshit.  However, there’s more to beta than just a label.  Smart companies stay in beta forever.

I’ll admit that when I first started Prefinery I was viewing it as a one-time expense for companies passing briefly through the beta phase on their way to public launch.  However, I’ve changed my thinking on this entirely.  Companies and products are constantly developing, changing, iterating and improving.  Prefinery is at the intersection of product development and customer relations and will provide value no matter where in the life-cycle a company may be at a given moment.

Use the beta label in moderation, but never move out of beta mentally.

Welcome to Prefinery’s official blog, written by me, Justin Britten, the founder of Prefinery.  I’ve carved out this slice on the interwebs to talk about building, discovering, and using awesome software, along with my experiences in launching a web startup.  Oh, and I’ll also talk about Prefinery product-related news, updates, features and whatnot.

I love new technology.  I love tinkering, giving my very opinionated feedback and helping to define the features and direction of a product. I love the feeling of an emotional connection between myself and a hip company that (irrationally) makes me want to don a tie and blazer and go door-to-door evangelizing their wares.

There are tons of people just like me, waiting for you to build something awesome so that we may critique and shape it.  We are your beta testers and we are your most important users.  We are your biggest fans, your evangelists, and your superusers-to-be.

Prefinery is built around the idea that the beta management process is a cornerstone of success and that this first connection with your users is undeniably important.  Our goal is to help entrepreneurs nurture the relationship with their beta testers and to build excellent products collaboratively.