By Sheryle Gillihan

Every organization that is developing an app has hopes of becoming the next viral hit, but even great apps compete for attention amongst the distractions.

Moneythink, the established and growing financial literacy program for urban, low-income high school students, also had high expectations of adoption for its app. After completing the Moneythink Mobile pilot last spring, our organization, CauseLabs, reviewed the quantitative and qualitative results with Moneythink. Their initial response? Disappointment. But the CauseLabs team saw something different. We saw the pilot program as a success.

Analyzing the Right Results

It happens all the time. Organizations that are building apps for the first time set the bar too high. Having years of app development experience, CauseLabs knew better than to expect astronomical results and saw success where Moneythink did not.

Consider some of the challenges of mobile engagement: Mobile users are selective when downloading apps due to limited device space. Moreover, studies have shown that, no matter how many apps are installed on a device, users only open an average of 20-30 of them per month. Even useful apps fail to hit 100% engagement.

Take email as an example of a useful app. For some of us, checking email is a daily to-do and behavior that has caused mobile email clients to far surpass their desktop counterparts. Yet, despite prevalent email use, fewer than 20% of email client users were active in the last 12 months, according to an IT Business Net article from earlier this year. Gmail, topping the charts for email usage, showed that only 11% of users were active over a 90-day period.

We don't expect to put a number as low as 11% on reports when we set new app engagement goals, but perhaps it is more realistic. Active use of an app from any user group (outside of paid staff) indicates that it offers something of value. When evaluating our work at CauseLabs, we look through the lens of the 1% difference. Not everything we build will be the next overnight hit, but we build tools to create impact. In the case of Moneythink Mobile, we are introducing financial literacy to the next generation of leaders. If we get Moneythink Mobile in front of 100 students and impact 1% of them in two months, what happens when we reach 1,000 students over a year?

Of the students in the Moneythink Mobile pilot group, 80% downloaded the app and 34% interacted with it, while 4% completed all nine challenges. This percentage may seem small, but these are power users. Moneythink can grow their user group and start to see the 4% impact increase over time.Classroom Testing Reveals Engaging Behaviors

Taking this more realistic perspective on the quantitative numbers into account, Moneythink also analyzed its qualitative results.

Some of the best results come from watching users interact with a product. After going through the human-centered design rounds with IDEO.org before the pilot, we landed on a challenge-based app with basic gamification incentives, as we previously described here on Core77, in an article about the app-building process.

After building the Android prototype in less than eight weeks, we took it into classrooms where we could observe how students interacted with the app and listen for feedback. We saw where they got stuck, and we also heard direct feedback about where the app should go next. Building an app to engage high school students was a lesson in the social user experience.

The pilot program validated two crucial items:

1. The app needed to be cross-platform compatible. Because the pilot was limited to Android users, participants were frustrated over excluding non-Android friends.

2. Students engaged more with photo-based challenges. This made so much sense and had even surfaced during earlier discovery sessions, but it was important to test the various challenges to test our assumptions about photo-based apps.

Given the results, Moneythink was confident in moving forward with its new curriculum, and not surprisingly, all of the challenges for the next phase are photocentric.

Intuitive and Inclusive

Our pilot observations also resulted in changing the structure of Moneythink Mobile. During the pilot, a few students accidently selected the wrong classroom. Since there was no quick way to remedy the issue, they got stuck in the incorrect classrooms for awhile. This turned out to be a happy mistake. We found the students liked to engage with people they didn't know. Activity went up during this time, and we reevaluated the decision to limit visibility of students' posts to their classrooms. Other successful apps allow interaction between strangers, and after some research, we decided to include this feature as a pilot for this year's program. Now students can explore what other students are posting so they can draw inspiration, interact, and feed on their generally social nature.

Students with iOS devices were another missing component during our pilot test. While Android was prevalent in the user demographic, we heard feedback that iPhones users were more prolific on their devices and more active in the Moneythink program. This may or may not be accurate, but we intended to develop for both platforms if the pilot showed signs of success.

We started building the iOS app in July and in a rapid race for this school year, launched eight weeks later in September. It is a version 1.0, so the user interface is where the Android pilot app was in the spring of 2014. It will be an interesting test to see how students respond to the differences between the more complete Android and prototypical iOS versions. Will they overcome the user interface to engage with their friends?

Because we want as many students as possible to participate in the program, the app will be available this fall to Moneythink students on Android's Google Play and the iOS App Store. In the next 1-3 months, we'll gather more feedback and analytics. No doubt, Moneythink has the potential to bolster financial literacy through its teen mentoring program. The big question we still hope to answer is whether Moneythink Mobile promotes ongoing behavioral change towards better financial habits and enhances the opportunity to impact more people in a more lasting way. If the answer is yes, then Moneythink Mobile could also reach new and different audiences, changing the landscape of how financial literacy is delivered to communities.

Sheryle Gillihan is Director of Partner Solutions for CauseLabs, which builds tools to impact people.