ArticlesBlog

Social Impact: Elizabeth Fowlkes

Social Impact: Elizabeth Fowlkes


Elizabeth Fowlkes: Before I sort of dive into
Boys and Girls Clubs and what weíre doing Iím just going to tell you a little tiny
bit about myself and my background which is that at one point in my career I did pretty
much a complete 180. Before I decided to go into the world and become somebody who was
interested in numbers and data and measurement and all of that stuff I started in the theater.
I did not last. I was not good. But once when I was performing I had an occasion to perform
in front of Mary Steenburgen. And after the performance was over I had lots of people
come up to me and ask and say, you know, how was that? You were you performing in front
of Mary Steenburgen. Were you so nervous? Sheís such an expert in acting and all of
that. And it was a little nerve-wracking Iíve got to say. Probably being up here in front
of all of you to talk about Boys and Girls Clubs of America in front of Roxanne Spillett
whoís such a leader and has such a long tenure in our organization might be a little more
nerve-wracking than that. So I hope to do a good job here in front of Roxanne for you
today. And I want to talk to you about the journey that Boys and Girls Clubs has been
on over really just the course of the last three years. Weíre a large organization with
a very long history. And about three years ago decided to really embark upon looking
at the way that we measure outcomes across out network. And we have really a very large
footprint. Iíve got some information for you up there on the screen, so that you can
get sense of the scope and scale of what we call our movement. And when I say the movement
which was a term that sounded really weird to me when I was interviewing; it sounded
a little bit sort of cultish. When you say the movement, what we mean by the movement
we mean local Boys and Girls Clubs sites, 4,000 sites across the world where young people
are served, where weíre actually providing services to young people. We mean the organizations,
about 1,000 organizations that operate those sites. Those are each independent 501(c)(3)
organizations. So their own board of directors. They make their own decisions. They have their
own governance. Those sites, plus those organizations plus the national organization and we provide
training, we provide technical assistance, we do program development, we do fundraising,
we do a whole host of things for the local Boys and Girls Clubs sites and organizations.
So when you see sort of the scope, when weíre thinking about coming up with a collective
way for us to measure our impact and to measure outcomes across Boys and Girls Clubs of America
weíve got a pretty big job. Now, one of the things you donít see up there necessarily
is not only are we large in terms of our size but in terms of our diversity. Weíre a very
diverse movement. So we may have some Boys and Girls Clubs organizations where thereís
one 501(c)(3) supporting one club site. And they may have an operating budget of less
than $500,000 a year. We have other Boys and Girls Clubs organizations that serve more
than 30 sites and have operating budgets of more than $5 million a year. So we are working
with a huge variety of different types, different sizes of nonprofits to help them do this important
work of measurement. And ultimately, weíre doing that in order to impact these young
people. I mean the way that weíre thinking about it– now, we do work in research. We
certainly work with third party researchers to take a look at what weíre doing. We do
work in program evaluation. We provide curriculum based programs to our clubs. But what I want
to talk to you about today is really our outcome measurement, the work that weíre doing in
order to drive continuous improvement in clubs. So putting together a system, putting together
in places both from the technology perspective to the indicator development, to the actual
ways that we measure these things. So that on an annual basis we can be measuring things
with consistency and credibility across the movement, so that local Boys and Girls Clubs
organizations and the national organization can be doing goal setting, can be doing benchmarking
and can be driving the continuous process of improvement. Now, hopefully in getting
all of that data itís going to enable us to do some even more research and evaluation
work that weíll get to down the line. But weíre really talking about continuous improvement
so that we can demonstrate collective impact, get better at what we do, raise more money
to serve more of these kids. So when we start to think about how to do that across a national
network weíve got to think about what are the barriers for our local organizations.
What are the barriers for that one organization, one club site, $500,000 a year organization?
What are the barriers for the very large organization? And Iíve listed a few of them here. Now,
many of you are probably very familiar with many, many of these barriers. I want to touch
on just a couple of them that I think are really particularly critical. One is the competing
reporting requirements. Iíll tell you a quick story. I went to visit a local organization,
a rural organization, one of these single site, single organization clubs and I sat
down with their person who was in charge of their outcomes measurement. Now, this is a
person who has a fulltime job with the organization, which they didnít have very many, maybe three,
four people fulltime. This one person her job halftime to sit at the front desk and
stand kids in, front desk responsible for knowing the kids, saying hello to the kids,
greeting the kids, standing the kids in and part time sheís responsible for all of the
data across that organization. So sheís got two hats. Not too bad. Many of us wear more
than two hats on any given day. When I sat down to start to talk to her about what she
was measuring, about what were the things that her organization was really looking at
she very proudly pulled out a printout for me. And this was a printout from an Excel
spreadsheet. It was at least 15 pages stapled together, very small, Excel rows. And on each
one of those Excel rows it listed one indicator, one thing that they were measuring in one
column and another column said when they were measuring it and in another column said which
funder they were reporting that indicator to. So three cheers for organization especially
for somebody whoís got half of her job as sitting at the front desk and actually working
with young people. Just scanning through that first page, just the first page of that Excel
spreadsheet, there were at least three things they were measuring for three different funders
that were substantially the same thing. They were getting at substantially the same outcome
for a young person measured in a slightly different way, on a slightly different timeline
for a different funder. So thatís one of the challenges for us as a national organization,
for us as our own nonprofit, and then for us as a network supporting all of these local
organizations is to figure out– I mean Scotland has one page. How can we simplify and be crystal
clear on what we want to measure? And if weíre going to do what Roxanne mentioned earlier
in terms of moving from outcome intended in order to outcome driven we need to be very
specific about what outcomes weíre trying to drive. Be very specific about how weíre
going to measure those things. And weíve got to be able to come to the table when we
sit down with our funders and say this is what weíre going to measure. This is why
this measure matters and here is the research that backs up that this indicator is related
to these outcomes that weíre affecting. And Iím going to be able to give you the data
on this, using a reliable method on a consistent basis because this is my plan. That gets back
to that question of leadership that we talked about earlier in terms of the person in charge
of the organization being the one whoís championing those measures and knowing whatís on your
one page. So part of what weíve done as a national organization and Iíll talk a little
bit about this is to figure out what is our one page. What is our list of indicators?
And then how can we help local Boys and Girls Clubs to measure those things consistently
and credibly. The other thing that I think is kind of an emerging area, maybe not so
emerging for some people, maybe more for others in terms of these competing demands, is competing
demands around technology. Iím not going to talk a little about collective impact.
I wonder if anybody will over the course of the next two days in terms of the move towards
people getting together in local communities to nonprofits who might have previously been
competitors or not working closely together, getting together to affect change together.
And a big part of that has been common outcomes, common data systems. So the challenge that
some of our clubs and other nonprofits find themselves in is thereís this competition
then between what Iím calling– Iím still working on coming up with a concise way to
think about this and to talk about it sort of your horizontal accountability within your
community, who do you need to share data with? Who do you need to work with to measure common
indicators within your own local community and your vertical accountabilities? So for
a Boys and Girls Club how are they able to participate in the network of Boys and Girls
Clubs across the country to gather best practice to measure things consistently, to learn from
each other as part of a community where we may be diverse across the country but we do
a lot of similar things. So thereís learning to be had there by measuring and sharing outcomes.
But also needing to participate across the community that theyíre part of, to be part
of that. And what happens when each of those things comes with its own data system. Enter
your data into my system and I will produce these outcomes for you that will manage your
reporting requirements. And itís not just local communities and national networks. Itís
different funders requiring different systems. And so this is something that more and more
is becoming a challenge for local organizations where theyíre either having to choose which
network theyíre going to be part of or theyíre having to do double or triple data entry in
different systems. And the more data we are entering, the worse the data quality is. So
this is a very real challenge for local organizations and I think itís a growing challenge that
weíre having here. Now, in addition to taking into consideration our local clubs and whatís
going on for them, what are their challenges and barriers in this area, of which, I only
touched on a couple we also have some of our own unique challenges as the national organization.
First of all, as I mentioned, the scope of our movement, thatís a challenge. How do
you design a measurement system that is going to be useful and helpful for the smallest
organization and the largest, the least sophisticated and the most sophisticated? Weíve got this
issue about local data systems. And in our case, we have very limited ability. We donít
have the capability to go out and require anybody to use a certain system. Very limited.
In some instances we can do a little bit of that for specific projects. But for the most
part this is trying to get a movement that is as large as ours to do something because
it is helpful to them and is a good idea and it will collectively help us without having
a very strong lever in order to do that. So these are the challenges that we sat down,
spent some time looking at, thinking about and our solution to it just is our national
youth outcomes initiative. And the national youth outcomes initiative we knew that we
needed to design this thing. We had a name for it and now it needed to have substance.
It had to be easy for local clubs to participate in so they would opt-in. It had to be flexible
so that they could one, participate in any of these other networks that they needed to
be part of. And two so that we could provide flexibility for local measures for locally
reflectively things that individual organizations would want to be measuring. We had to make
it compatible so that they could continue to participate in other networks. And, of
course, it had to provide meaningful information back because we needed for our local organization
to choose to opt into this. So it has to add value. It canít duplicate something else
that theyíre all ready able to do with perfect ease. We needed to add value. So Iím just
going to take you through five steps that have gotten us to where we are. Weíre still
in the middle of this journey. We are nowhere near the end and these really arenít perfectly
linear steps. But just as a way of sort of organizing our path through this today Iím
going to take you through five difference pieces. So the first thing that we needed
to do was really think about what are we going to measure? Thatís where we have to start.
And so starting with the end in mind, we first went to our mission and Roxanne talked about
our mission earlier and it is this big hairy thing. We want to help to produce productive
caring people. So thatís hard to measure. And we needed to be able to talk about what
we were going to measure in an easy way. We saw a simplified version of a logic model
earlier. Jeff, showed us a simplified version of theirs. Ours is even more sort of simple
and user friendly. This is more of a theory of change for us, a visual depiction of how
we do what we do and itís simple on purpose so that we can very easily help all of those
thousands and thousands of staff people at Boys and Girls Clubs across the country to
understand and articulate how what they are doing on a daily basis is contributing to
outcomes for young people. So when you look over on the far left hand column, this is
how weíve articulated what that mission is in terms of outcomes. There are three that
we are looking for. We are looking for academic success. That means high school graduation
with a plan for the future to us. Thatís how we weíve articulated the academic success
piece. Then we also have an outcome around the area of good character and citizenship
and around health lifestyles. Thereís a lot of other important pieces on this theory of
change that Iím going to come back and talk about later. But for a minute letís just
look then at what weíve done in terms of trying to think about even when we get to
academic success we took the mission and we broke it down to academic success. Okay, thatís
great. Now, we got down to high school graduation. We canít afford to wait until our young people
are 18 to find out if weíve achieved that. So the indicator development piece was a really
important part of this process. So what we did is we went to the research Roxanne mentioned
earlier. We worked with Child Trends to do this. We went to the research and we said
what are the things that research indicates are related to later high school graduation.
What are the things that we can look at for an eight year old? What can we look at for
a ten year old? What does research tell us is meaningful for a 14-year-old, a 15-year-old,
that are things that are then related to later high school graduation. That was one lens
we looked at it through the first one. What does research tell us matters? Other lenses
we looked at it through what has communication power? Because ultimately weíre going to
want to use this information with various audiences. And then we also looked at what
can we reasonably expect for Boys and Girls Clubs to be able to collect. Some of the indicators
that ended up on our list are harder and some are easier to gather. So when we talk about
our indicators with clubs, weíve got a list of them that we call advanced indicators and
weíve got some that are basic. Weíre trying to get everybody to collect the basic stuff
and are providing them with the mechanisms to do that. Some of the advanced things are
just too important. They are too critical in the research and they have too much communication
power for us to ignore. But we realize that not every club is going to be able to collect
all of this because in many cases, when you look at some of these academic success indicators
itís going to require very strong relationships with the school district in order to be able
to get meaningful data on these things. And so we recognize that. And so we call those
advanced and weíre going to work towards helping organizations build those relationships.
Some clubs all ready have it. We also, you can sort of see this spectrum that I put up
here. Ultimately, what weíre trying to impact are those longer term outcomes. What weíre
measuring and what weíre focusing on measuring with our national youth outcomes initiative
are sort of the intermediate timeframe variables. Historically, when weíve looked at measuring
things weíve looked at individual programs, so curriculum based, time limited programs.
Weíve looked at measuring those. So the sort of pre-test, post-test, the beginning to the
end of an eight week program, that kind of thing. Those are valuable measures. Those
are going to give us sort of short term outcomes about changes in knowledge, maybe, if itís
a knowledge-based program. We decided with this that we wanted to focus on more intermediate
things that we might see, we might measure on more of a one to two year kind of a basis.
That weíre going to be maybe more directly related to the long term outcomes that we
were looking for. But what we then have to give up is a degree of certainty about how
much of a factor it is the club that is playing into that. And we know that. And so thatís
another piece that weíve had to consider in terms of some of the things that we measure
and some of the ways that we look at this data. So those are some of our academic success
measures sort of down the middle there, some of things that weíre looking at. And the
other outcome areas I just wanted to share with you some of the things that weíre looking
at in those areas. For good character and citizenship we look at community service.
We look at actual hours. We also look at self-reported participation in service. And for all of these
indicators, one of the things that we did for our clubs in order to make this easier
for local– for our local organizations to take it is that weíve written up for them
a very short, simple explanation of the research behind each of these indicators. Three to
five sentences for each indicator with the citation so that we can start the process
of educating local organizations about why these are the indicators that were chosen,
what is the research basis on that, so that not only they can become educated themselves
but so that they can become advocates for these indicators. So that when they are sitting
at that table across from a funder and they want to be able to say this is what weíre
measuring in this area. They have the language to go along with that in terms of why we chose
to measure these things. So we come back to this picture, to this theory of change, what
we call our formula for impact because weíre not just measuring the outcomes. The outcomes
are certainly a critical piece, certainly a critical piece to the impact. But we need
to be able to measure other things in this diagram in order to actually sort of really
analyze what weíre seeing in the outcome side. I mentioned earlier that when you look
down the middle column here underneath the outcome drive club experience, we take the
young people who need us most, thatís part of our mission, weíre always driving to work
with the young people who have the greatest need. Then that middle column really articulates
what are the things that the club provides. Targeted programs is the third thing you see
down there and thatís those curriculum based programs that are going to be focused on one
of these three outcomes. Theyíre going to be a specific program around nutrition or
a specific program around mastery of skills and academics, things like that. Thatís a
really important part of what we do. But it exists within a much bigger context of what
we call the outcome driven club experience because young people donít come to us, sit
in a classroom to take a program and then leave. Typically, the program is embedded
within a larger day where theyíre having interactions with caring adults. Theyíre
having the opportunity to express themselves, to do lots of different activities. So there
are these other pieces that are really critically that we want to be sure are included in our
measurement strategies. So these five key elements for positive youth development, these
are five things that we work with the search institute to sort of validate our important
youth development principles that we measure in our Boys and Girls Clubs. We train on these
and we measure them. And itís things like first and foremost are young people safe?
Do they feel safe? That the club is a safe place to be? Is it fun? Because if itís not
fun theyíre not coming back and theyíre certainly not going to get into that targeted
program you want to put them in. Are they building meaningful connections with adults?
Do they believe that adults have high expectations for them? Do they believe that theyíre provided
with opportunities and expectations? These are sort of fundamental foundational building
blocks of youth development that are really critical. And we need to be able to measure
those to see to what extent they are actually being carried out to help us identify which
clubs are doing a really good job of it so we can figure out what theyíre doing thatís
producing that. So thatís a piece that we need to be able to measure. And then as we
move down high yield activities those are activities so we call those sort of fun with
a purpose. Those are quick one off kind of activities that are targeted towards those
outcome areas. And then attendance, we did some work with public/private ventures to
take a look at attendance and how it relates to outcomes. So we know that there are certain
attendance thresholds at which young people who attended those thresholds are achieving
outcomes at a greater rate than others. So weíve got certain thresholds that we look
for in our attendance data based on the research that we did with public/private ventures.
And so thatís another sort of indicator that we look at. So we thought about building out
this measurement system, building out a way that we were all measuring things in a common
way. There are eight different demographic characteristics that we capture. The system
that we built is on the individual member level so we can track individual kids longitudinally
over time. And these are the characteristics that weíre tracking on them. So we bring
in tenure at the club, so longevity of membership at the club and frequency of attendance as
member level characteristics so that when we get to the outcomes data we can cut all
of the data in these various ways. Now, clubs participate at different levels. Some clubs
may not be capturing military household. It may not be meaningful in a certain community.
So clubs opt-in or opt-out of providing the pieces of data that are relevant. Now, there
are certain pieces that we say that are pretty basic that everybody pretty much does, as
you can sort of imagine what some of those are. And there are some that people opt-in,
that clubs will opt-in or opt-out of depending on their community. And the other things that
we measure are those club experience indicators so that we can look at that data. And then
participation in specific targeted programs. So weíre looking at whoís participating
in which of those curriculum based programs and then just generally club attendance. So
thatís how we came up with the indicators, sort of starting with the research, thinking
about what clubs could do and then bucketing them in terms of whatís reasonable for everybody
to do, and whatís reasonably only for some clubs. And those are then areas where we can
work on training to help clubs get there. The next thing that we needed to think about
was where to put all of this data if wanted to track it on an individual kid basis and
start to build out a longitudinal data system. Weíve got to put it somewhere. We did a lot
of the thinking, I guess, Jeff, like you guys did and for our organization, for a host of
reasons we decided not to build a single system that everyone would be required to use. And
thereís a lot of reasons for that. We took a look at the time when we were starting this
and about 80 percent of Boys and Girls Clubs organizations were all ready using a database
where they were recording a lot of that data that we needed all ready. And so what we did
is we worked and we looked at who those vendors were and we built out a partner vendor program
where we approached those vendors and we said weíd like to partner with you on this thing
weíre doing called the national youth outcomes initiative. Weíd rather not get all of our
clubs to join a single database and leave you. And instead will you be our partner in
this and will you help us to build a technical bridge, so that the data that all ready exists
can get a double use. It can be used in the way that the club is all ready using their
local data system. And then the relevant pieces of data can be piped over into our national
database on a regular and automated basis. So this is what it looks like now. Clubs enter
data into their existing member management system. Thatís MMS up there, thatís how
we abbreviate it, doing what theyíve always done. This is a big selling point for clubs.
We wanted to make as little new data entry as possible. We attend to leverage to the
greatest extent possible work that was all ready being done. So clubs are doing what
theyíve always done. Then after they set up their system and they tell us which variables
they want to transmit over and they do a bit of a mapping process. So we might have some
clubs that are collecting– I mean weíve got a club thatís collecting 22 different
race variables. They capture just about every bifurcation of multiracial as you can imagine.
And what they do when they set up their data to transmit over to us is they map then the
elements to our list of race categories in that example. They map that data over and
then data gets submitted to us. We get attendance data weekly, daily attendance that comes over
at the end of the week. And then we get most of the other data on a monthly basis. The
database, our NYOI database then calculates the indicators that we want to be looking
at so we know that theyíre being calculated in a consistent way rather than saying please
calculate your average daily attendance, choose your favorite formula. We are then standardizing
the way that the formulas are done and weíre taking the raw data, calculating those indicators
and then piping that back out into reports that are available to our clubs over the Web.
So thatís how we set up to utilize thatís all ready existing. Now, thatís really–
when weíre talking about the data thatís in the club system thatís mostly basic information
about their members, attendance data, program attendance data, maybe some community service
hours and for some of those really advanced clubs they are getting some other more advanced
outcomes information. They might have some data on fitness. They might have some data
on school attendance, state test proficiency, things like that. But for the majority of
our basic indicators, weíve developed a national outcome survey so that we are providing the
tool to clubs to collect the data on the bulk of our what we consider to be our basic indicators,
so that across the movement folks are using the same instrument, so that we can really
do comparison of the data. So our national outcome survey is administered annually in
the spring. Itís a member survey so weíre actually surveying members. It is confidential
and voluntary. We follow a consent process. We work through an IRB each year. And we have
surveys for members that are nine to twelve and another one for members that are ages
thirteen and up. We do it in both English and Spanish. Weíve tried to make this as
accessible as we can. We do get requests for other languages. We would love to be able
to do that and weíll work towards that. Our preference is for folks to use the Web-based
survey. Thatís for a lot of reasons. It sure makes data cleaning easier, as everybody in
this room can appreciate. But it also allows us to provide an audio component for our low
literacy, young people without adults having to read it aloud. And so that young people
can work at their own pace. Itís also a lot less expensive. But we are– have been offering
a paper booklet option also for clubs, where young people put their surveys into a ballot
box, a tamper evident ballot box. That gets mailed back to us. And then we, working through
a vendor do all of the scoring so that clubs arenít doing that. Again, so that the clubís
job is as easy possible. So what we are minimizing the amount of time they are spending on data
collection, maximizing the amount of time they are spending interacting with young people
doing the hard work of youth development. And then we are taking on sort of the complexity
of doing all of the scoring on the back end. So then our survey data gets brought in and
married up in our database at an individual level so that we can then also calculate indicators
on our outcome data and include those in our reports that go back out to clubs. So we really
went to the practical part here in this talk. Thatís very much sort of the nuts and bolts
of how we developed a system to do this. But the other pieces of this that are so important
are the training and the data use because, again, this is only successful if everybody
knows what theyíre doing and they are putting the right data in the right places using the
right methods. So training is huge. The first thing that we really had to tackle with training
was awareness and buy in. We got this national initiative now, you know, we built. Now, we
need clubs to come and use it or else itís not going to do us any good. We piloted this
thing in 2011 with a small number of organizations. We really wanted to test out the survey in
the first year, do the validation on it. And then starting in 2012 really started pushing
to get clubs sort of first connected so that their databases submitting over and we got
that technical connection made. And then doing that annual survey. Weíve got 475 organizations.
Thatís about 50 percent. Right now, weíre only working with our non-military organization.
So thatís about 50 percent of our non-military organizations that are connected and will
be submitting data over Friday night, every Friday night. Weíre going to have about 90
percent of our organizations by the end of the year this year. Weíve all ready got most
of those in the pipeline and are scaling up pretty quickly. For our national outcome survey
you can see the numbers there. Weíre surveying really a huge number of young people. From
a national perspective, we donít need to survey that many young people in order to
get some really valuable data that we can use. But we want this data to be valuable
for individual local Boys and Girls Clubs organization that theyíve got enough representation
in their numbers so that they can meaningfully use the data at an organization level. And
ideally to be able to compare club site to club site so that they can see whatís going
on with young people that are served in different locations. So that was hurdle one with training
was just this thing exists, come do it. We think itís going to be valuable to you. Hereís
how. Hereís how you prepare your data. Hereís how you clean your data. Hereís how you select
what should be sent over. Hereís how you get on the website to look at your data. One
of the things that this has allowed us to do having the standardized method is itís
allowed us to standardize a lot of our training. So just some of the trainings that we do are
up here on the screen. The measurement strategy training work that we do really is about sort
of talking to leaders about how they lead measurement. And how they think about being
strategic, about the indicators that they choose to use, about how they are then making
sure that that data doesnít just go sit in a filing cabinet on a server but it is actually
part of their way that they operate their business is looking at this data both for
internal improvement and in terms of talking externally to donors to raise more support.
Weíre able to do relatively standardized trainings around data collection and storage.
We have to customize those to the vendors because clubs are still using different vendors.
But now itís a finite number. We know which vendors they are and we actually work in partnership
with our vendors to develop those trainings so that itís not generic training. Please,
go clean your data but we can get very specific. You should clean your data. Hereís the button
you push to run this report to help you find duplicate members in the system and delete
them. We can get very specific in our training resources which is so helpful to folks on
the ground working with this information. And then, of course, we can– what weíre
working towards now that weíve got data in the system is trainings around how do we use
data in order to drive impact? How do we think about continuous improvement? What does that
mean? How do you think about that in terms of annual planning, strategic planning and
all of that? So training becomes a big piece. Not as big as using data, though. And that,
I said, you know weíre early in this journey. In some ways it feels like weíve really walked
a long road all ready. But I really do think weíre just getting up to the starting line
because now weíve got data and we need to begin the hard work of really using the information.
You know, I just put up a smattering of the different things that we want to be using
data for, things Iím sure that you can imagine that folks would want to use data for. Locally,
I think everybody is using data around– when I say everybody I mean the national organization,
and the local organizations and the sites. Our hope is that everyone is using data around
sort of goal setting, benchmarking finding other organizations that are performing more
strongly, absolutely around fundraising. For us, as the national organization weíre really
then working to use the data to find sort of the bright spots, the outliers, where are
the really strong organizations. If weíre able to determine that, a particular type
of club, a club in a particular facility, one of the things we know for example is that
when we think about the club experience, those key components of some of the foundational
stuff around youth development, some of our clubs that have the hardest time with that
are our school based sites. And that probably has a lot to do with the facilities and external
things that are going on there. So weíre going to be looking then for who are the school
based sites that are in the 90th percentile in that club experience? What are they doing?
What is different about the way that they work with their school district? What is different
about the way that they do things with their facility, with their space, with their training
and with their staff. Thatís going to be a lot of the work that weíre working on at
Boys and Girls Clubs of America. And then to sort of help to codify what does that club
experience look like and how can we then get more specific with training around that. I
thought I would give you just a couple of examples of what this actually looks like.
Iím a visual person. So this is an example of just a type of one of the indictors that
a club would get in their reports. This is around the area of recognition. To what extent
do young people feel like staff? Recognize them when they do a good job. And you can
see that what the organization gets is they get their own numbers. And then they get comparisons
to the state figures, all of the Boys and Girls Clubs in the state that are participating,
all of the Boys and Girls Clubs in the region and then national information. So they can
see how theyíre doing. Theyíll be able to track themselves over time but then also to
see where are they relative to other Boys and Girls Clubs so that we can start to target
in any given year. We canít work on everything. So how do we choose? How do we choose what
are the focus areas and building out the annual plan? Being able to do this kind of comparison
work is helpful. One of the things that we did when we built out our indicators and specifically
when we built the survey is that we used existing measures to the greatest extent possible.
So when we looked at many of our healthy lifestyles indicators, we borrowed items directly from
the CDCís youth risk behavior survey so that we would be able then to not only do internal
comparisons of Boys and Girls Clubs to other Boys and Girls Clubs but we could compare
Boys and Girls Club kids to national data thatís available through the CDC. So what
you see here is an example for just four of our indicators of data from our 20/20 national
outcome surveys, specific to ninth graders and how they compare to ninth graders on the
CDCís survey. So thatís another thing thatís available to clubs through the tools so that
they can have access to this type of information. Another way that weíre able to do comparisons
is to use internal benchmarks. For a lot of the indicators that we have, wherever we could
get an external data source we did. But for a lot of the things we want to look it just
doesnít exist. So because we gather tenure and frequency of attendance as individual
member demographic characteristics then we can break indicators down and we can look
at what does this outcome look like for young people who attend the club with high frequency
versus what does it look like for young people who attend the club with low frequency? What
does it look like for somebody whoís been a member of the club for two or more years
versus someone who just joined? So weíre getting some internal comparison groups within
our own population of young people who sort of selected them to come to the club. And
then weíre able to use those things to do some other comparisons. So thatís just another
way weíre looking at it. One of the last things I thought I would show you that we
enjoyed looking at in our data is we took a look at how the club experience specifically
those five key elements comparison to the outcomes that weíre looking for for our young
people. And one of the things that we found is that when we look at optimal club experience
you saw that on the recognition slide I showed you earlier. Optimal club experience indicates
that the young person demonstrated through answering many questions that they are having
an optimal experience in a given area. So they are clearly having fun at the club. They
clearly feel safe at the club. They clearly have strong connections to an adult at the
club. We look at that in terms of optimal. The other end of that is insufficient. They
clearly do not feel like they are getting those things that we know that they need as
the good building blocks. And then weíve got some kids that fall in the middle, where
we canít really tell. They answered the questions in such a way that they didnít clearly fall
into either bucket. So we see that for members that are reporting an optimal club experience,
they are getting more of those positive outcomes that we want to have than the average club
member. So then thatís sort of the next step for us is to go, okay, what is it about those
clubs? What is it about the club experience that is positive? How do we replicate that?
That becomes then just another road for us, an inquiry for us to start to explore this
data as we go forward. And this just shows that on the other side. These are indicators
where we want less of the behavior, not more. And we see a consistent pattern here in terms
of the relationship between the club experience to those indicators. So I just threw up a
smattering of additional findings but I do want to move pretty quickly into having a
dialogue with you all about this work. Some of these additional findings really are just
a jumping off point for us. We did this for the things that we found in terms of race
and gender and pretty much every way we could cut it we started, but again weíre just at
the tip of the iceberg in terms of looking at it. And so when I think then about whatís
next for us certainly continuing to delve into a lot of these questions. I mean certainly
the first year, year-and-a-half of data collection has just raised more questions than itís
answered for us to dig into the information. So more work with the data is certainly on
the horizon but the other piece for us is really thinking about the cultural shift and
about the leadership piece like Roxanne talked about earlier and being sure that the leaders
in all of our organizations and the leaders of our own Boys and Girls Clubs of America
organization are looking at data, using data to make decisions, to set goals and to continually
improve, to really focus on that continuous improvement. And Iím not brownnosing. I did
not know they were going to have this book on the table when I put the presentation together.
But I did pull a quote from Mario Morino out of here because I do think this really captures
where we are. Weíve got a system in place to get the numbers. We still have some work
to get everybody there. But the system is there to get the numbers. Itís that cultural
shift to be sure that weíre thinking about the numbers, and really thinking about in
terms of the way we manage. That is where weíre going to be moving going forward. So
with that, I want to ask what questions you have for me.
Bill Cleveland: Thank you very much for a very nice presentation, Elizabeth. Iím Bill
Cleveland [ph?]. And Iíve got a question I had earlier for presenters and that is with
the system that you set up have you been able to influence donors to say all right, instead
of these 15 pages of stuff, of measuring our organization we think even better, can you
proactively say we know how to measure our organization. Weíre going got give you the
data thatís really appropriate, rather than a smattering of things thatorganization
. Weíll give you something. And then also have you found your programs are
being changed because you decide weíre wasting our time doing this and we can better serve
our kids by doing something different because of the outcomes weíre seeing?
Elizabeth Fowlkes: Yeah. So on the first question Iíll give you two examples of that. And so
you asked about have we actually seen the ability to go to a funder and kind of change
the way that theyíre thinking about this? And we have. And Iím going to give you two
different examples. I don’t believe, I will say, I donít believe weíre going to get
to 100 percent dictated by the nonprofits. But if we can hugely shift– you know, if
we can move from a 90 percent dictated to a 10 percent dictated or even from just a
little bit of a shift I think is going to make a world of difference in terms of what
folks on the ground are experiencing. So Iíll give you two examples. One was with one of
our local club organizations and theyíre United Way and they had historically for years
and years had this list of indicators from the United Way that they needed to measure.
They got connected with our national youth outcomes initiative, started using these measures
and they contacted me and said, ìWeíd really like to not be doing these measure for the
United Way anymore. Can you help us to frame up a message to United Way?î And I helped
them. And we used that document that I mentioned earlier where we wrote up the research for
each one of the indicators and we matched what the United Way had been looking for.
Hereís the thing, the piece of data youíve been looking for, thatís really getting at
this. Hereís what our measure is that also gets at that same thing and hereís the research
that goes behind it. And it was not very much work because we had all ready put together
this document for clubs to use movement wide, helped them to craft that message and their
United Way accepted it and they moved forward. As the national organization the first example
that comes to mind for us didnít go so easily or so quickly. We sort of had the same conversation
with the funder and we were early enough in the process of building this thing out that
they, I think, understandably said weíd still like for you to report what we want. And so
what we did is we said, okay, weíre going to give you both. Weíre going to give you
what you want and weíre also going to give you the things that we know that weíre measuring
consistently and credibly and are going to be able to give you overtime. And where we
can give you external benchmarks because this funder was particularly interested in funding
some health outcomes where we had YRBS comparison data. And so we provided them with both pieces
of data for several years into the grant. And by the time it came around to the renewal
time we are only doing the things that we wanted to be measuring. So again I donít
know that weíre going to get 100 percent but we are starting to see some movement in
that regard. In terms of the program piece, we have not gotten to looking at specific
programs too much with this. weíve done it a little bit but we havenít started slogging
through the data enough to get down to specific programs enough for me to really tell, give
you a specific example of that but I really hope that we will. Yes, sir.
Chris: Chris.. Elizabeth Fowlkes: Thank you.
Chris:. So I have a few questions. One, is what do you wish all clubs knew about
this. Two, what are the organizational. It just seems like thereís a
lot of– thereís obviously going to be a relationship..
Elizabeth Fowlkes: You know, in terms of the board, I mean I think we still have so much
work to do in terms of awareness building around boards. I mean I think the thing that
I really wish board members knew was that this existed as a resource so that you could
be asking about it in board meetings and saying, as part of your annual planning and as the
leaders of the organization saying those questions and the conversations that you have about
sort of finance and board development and resource development are so, so important.
And if the starting point for those conversations can be this data about whatís going on with
the young people so that weíre targeting the resources that are coming in to the needs
that we can see in the population. We have started doing some board trainings. And was
recently talking to a colleague who just got to go out and do a board meeting and show
the board the data for the first time and he just got very excited because it just gives
some direction to the conversation. Always, weíre going to need to raise money but it
helps to give some real urgency and helps to make some decisions about where to prioritize
in terms of that. So I think just generally that this is here as a resource would be my
answer to the first. And remind me of the second?
Chris: Just what are the different club attributes that you are considering? Because.
Elizabeth Fowlkes: So we really when weíre thinking about the organizational attributes
we think about that at two different levels. So we look at attributes at the club site
level, the place where kids are being served. So thatís where we look at does a school
based site look different than a public housing site look different what we could sort of
more of a traditional club site, more of a community center kind of site and what are
learning about that? Because thereís so much variation there. And then also by the site
level weíre looking at rural versus suburban versus urban locations and how that factors.
All of those things all ready weíre seeing lots of– weíre learning a lot in terms of
that. Then from the organization perspective itself we look at budget size. I guess back
on the club site size we look at sort of community distress and concentrations of poverty. Weíre
also looking at we have as organizational measures weíve got something we call key
performance indicators. And those are indicators around– I mean everybody knows what key performance
indicators are. Ours are specific to board engagement, fundraising, strategic growth,
things like that. So weíre going to be looking at how those key performance indicators at
the organization level relate to these indicators as they change together over time to see if
there are things that weíre seeing as sort of leading indicators towards some of the
other things. So thatís work that weíll have to do as we build out more data.
Chris:.benchmarking forquestions.
Elizabeth Fowlkes: Absolutely. Lots, lots more to be done in that regard, for sure.
Yes. Woman 1:mentioned..
Iím. We fund two Boys and Girls Clubs.community impact focused.
The first round was to ask organizations to give us what impact indicators are going to
use? Well, we just sat down this week to discuss how across the board we were disappointed
with what we got. And kind of brainstorm around where we might go with this and how we would
improve and so on? So Iím looking forward to going back and saying, you know, letís
look to our Boys and Girls Clubs which are big investments in terms of our own United
Way and where they take a lead in this. So I found it very useful in terms of that
in my community. A question for you, both you and Roxanne a little bit earlier,
in a way that youíre sort of putting yourself to the side a little bit with the collective
impact work thatís around the common indicators and so on and youíre going your own way
cost to that? Do you really see yourself separate from the network?
Elizabeth Fowlkes: I donít see us as separate. I think I wanted one to put it to the side
today because I could stand up here and talk about here for an hour and we just donít
have enough time. But so I donít think that we are separate from it at all. But as a national
organization that works with organizations all over the world, really, it will be impossible,
I think, for us to align with any one sort of individual collective impact. So the best
we can do is provide support sort of from that vertical perspective so that we are measuring
things across Boys and Girls Clubs. And then have made that easy enough for clubs so itís
not too onerous so that they can still participate with their own collective impact in their
community. And hopefully, there is some synergy between the indicators. I mean I think a lot
of the things weíre measuring I mean these are the things that it makes sense to measure
in this area. But there are going to be organizations whether itís because of their collective
impact initiative or itís just because their community has a particular issue, theyíre
going to be organizations that need to measure things that are a little bit different, that
are something in addition. And so we just tried to make this as light weight enough
so that that would still be possible. And I think we are still trying to figure out
how to best support clubs in the collective impact world because it looks so different
all over the country. And, in fact, for some of our organizations theyíre so large that
they may have different club sites that are part of different collectives even within
their own organization and thatís a challenge. That technology challenge is huge there. Yes.
Art: Hi, Art, again. One of the things thatís scaring healthcare organizations right now
was mentioning is that there are regional trends in how people answer satisfaction
surveys. New Yorkers just answering more grumpily than the rest of us across the board. So to
do a fair assessment of them you would have to sort of adjust for those regional trends.
Have you seen those same sort of patterns in your data? And if so, are you making any
adjustments for this?Elizabeth Fowlkes: I would say generally we
see that across the country just working with so many organizations we see. I wonít say
which regions are some of the most grumpy but we do. We arenít making any adjustments
in our data for that. I mean itís one of the reasons I think clubs are more like when
theyíre looking at their own comparison data to be more concerned with the pie chart thatís
right next to their own, right. So if Iím the organization Iím looking at my state.
Or one of the things that we want to bring in though are similar organizations. So if
Iím New York City, I donít really care about the rest of my state. I want to compare myself
to Los Angeles and San Francisco and Houston and Chicago. So weíre going to be building
out sort of a similar organization so that it will allow us to look at things like just
show me clubs that are in my budget size and have other similar characteristics in that
comparisons. So thatís on our long backlog of things that weíre going to get to. Just
in our training then as people are looking. I mean we have sites. I mean youíre going
to where every single indicator they look worse than everybody else. And that could
be some regional differences. It can be some real problems that theyíre having in the
club. But itís their starting point. And thatís one of the things that has to be a
big message for us is this is not about being punitive. This is about continuous improvement.
So this is your starting point. And if your kids are grumpy, theyíre going to be grumpy
every year. So what weíre doing is weíre setting the baseline and weíre saying what
we want to see is the scores to improve no matter where youíre kind of starting out.
Man 1: Obviously in your statistic about an optimal involvement is going to lead to all
of these other beneficial impacts. Did you discover anything thatís surprising that
without this kind of survey activity the Boys and Girls Clubs would have never discovered?
Elizabeth Fowlkes: Thatís a really good question. Let me just say one thing first while the
back of my brain thinks on answering that one. You know, I donít know that we know
that itís the club– one of our questions is does the club experience cause it or does
it not. Is there some other factor? What is it? So weíre careful about making sort of
that causal relationship. We certainly hope and think that thatís the case but weíre
looking into it. I think one of the things we found was surprising to us was really some
of those differences around different types of club sites. You know, one of the things
that we found and this sounds a little backwards so hang with me, our young people in school-based
sites had some of the most positive outcomes in the area of academic success. And I think
we could kind of see that theyíre in that environment. Theyíre coming straight after
school. We also see that thereís a relationship between a positive club experience and positive
outcomes. And we see that young people in school day sites have some of the least positive
club experience scores. So now that seems– so thereís something going on beyond the
club experience in those school-based sites. One of the things that leads us to think is
if we can amp up the club experience and those school-based sites man weíll have a really
good– that seems like a really good thing to get going. If we can take whatís working
in schools and add whatís working in some of our traditional club sites in there that
seem like a really excellent opportunity. So that type of thing we saw some really exciting
results around some of the club experience factors in our public housing clubs that we
want to dig into. So some of those areas are some of the things where I donít know that
we would have known that otherwise. And it gives us some pretty clear direction then
in terms of what we need to go learn more about and what we need to start creating trainings
about so that we can improve. All right. I think thatís it.
Bernice Smoot: Thank you. Bernice Smoot. Elizabeth, really great presentation. I saw that you
measure for a number of things that are pretty critical and impactful to youth. And I saw
risky behaviors. But, in particular, do you measure around the issue of avoiding teen
pregnancy. And then also youíre measuring period in conjunction with YRBS, was it the
same timeframe for your program in this? Elizabeth Fowlkes: Yeah. Good question. So
as far as teen pregnancy goes, really what we measure is around sex, sexual activity,
condom use, the convergence of other risky behavior with risky activity. Weíre not asking
questions around pregnancy specifically. What weíre looking at is the behaviors that lead
to pregnancy. As far as the YRBS survey we do ours annually in the spring. YRBS only
does theirs every two years. So we do it– and we get our data out. So we had data for
our 2012 late June, we had that out to clubs. And at that point, the only data that was
available from YBRS was the– gosh that would have been 2009, YBRS data. So thatís what
we had to compare to so thatís what we were comparing to. And then as soon as the 2011
came out just a month or after that we then updated that data. So we get it as current
as the YRBS data is as current as we compare to. Thanks everybody for letting me come and
talk with you. I appreciate it.

Comment here