Methodology, Skills and Professional Benefits Participants Participants' Comments MEDICI Head of Training «The 12 Labours of Hercules» Report of the Reports – Workshops 1 to 4

Reports Previous Workshops

Fourth Workshop Report — 16 – 18 September 2014 — Štiřín (Czech Republic)

Module 6 – How to Evaluate Success

Introduction

Success evaluation is the key for film fund managers. It is a matter of accountability not only with regard to the politicians and other instances, but also as to their own management and decision process. Not only can it influence the subsidy level for a given project or producer but, in some cases, it can also determine their very eligibility, their evaluation criteria, the genres of the films for which support is sought, etc.

Or, in other words:

• Which tools can a film fund develop to evaluate its strategies and policy?
• How can best use be made of success evaluation results?
• What data should be used?

How does Telefilm Canada measure the success of Canadian films, and why?

Michel Pradier, Director Feature Film – Telefilm Canada (www.telefilm.ca)

See also:

• Michel Pradier’s presentation “Measuring Success – Telefilm Success Index” (PDF)
• Questions to Michel Pradier (PDF)

1. Reasons for changing the success evaluation of a film

We changed our success evaluation for the following two reasons:

• The Federal Government decided to give us 100M CAD a year to sustain feature films and attain 5% of the market share. Our market share has been below 5% since 2000 because of the vicinity of the US, and of the fact that English-speaking Canada is assimilated into US cinematography. US films make up 85% of the market share. Some years, like the year “Avatar” was released, that percentage jumps even higher.
• In our funding system, everything used to be calculated on the basis of the net national box office. On the one hand, this created discrimination and overlooked the achievements of many films. On the other hand, some producers were left with no access to automatic funding. However, box office grosses do not represent the totality of a particular film's achievement. It says nothing about selection and awards at the major festivals, nor about VOD sales and domestic or foreign platforms.

2. The new philosophy for success measurement

• We decided to evaluate films from three points of view: economically, industrially and culturally.
  • A film's commercial component (economic value) is allotted 60% of the points (40% relating to box office, 10% to domestic sales other than box office, and 10% to international sales). The big players in Canada that used to get 100% of the points based only their box office took to protesting. It took us a year and a half of consultations and discussions to reach the final model. Our first proposal was that only 30% of the points be allotted to the box office, but in the end we negotiated 40%.
  • Cultural component (cultural impact) is allotted 30% of the points. It covers international and domestic festival selections and awards.
  • Industrial component (industry health) is allotted 10% of the points. It represents the input of both the market and private investment in each film.
• The scoring has to reveal the totality of the value of a film. By aggregating the scores throughout a specific time period, we can evaluate the success of Telefilm Canada as a fund, and demonstrate how each individual production company performs on the basis of the score for each film they produce. Also, creative teams (producer, screenwriter, director) get an individual index for each film in which they are involved.
• We started applying our tool in 2010 by calculating the box office of all Canadian films for that year: it amounted to 24,6M CAD. The year 2010 became the reference year. We compare results under each heading: this helps us see where we scored well and where we underscored, and gives us indicators for new initiatives we may launch to improve things overall or in certain segments.

3. Data sources

• We have an aggregator (a device called MPTAC) that provides us with the national box office data on a weekly basis.
• Data for the cultural component, including festival selections and awards in Canada and abroad, are collected either by the festivals themselves four times a year (for the four prominent national festivals), or by external persons in contact with the international festivals (incl. Oscar, Golden Globe, etc.). We consider only the major awards (best film or best director). In total, we have 60 cultural events recognized by Telefilm’s success evaluation system.
• The industrial component is checked through the financial structure of the films we finance, by calculating public-private financing ratio. For the films we support, we demand that producers declare the entire revenue on an ongoing basis.

Challenges for the future

• We would like to measure our film's social impact as well, and how they resonate with the audiences, but we still lack any device for collecting data of that sort.
• Increase the frequency of distribution reports. All the information on domestic and international sales (trade level) is collected through the distributors’ sale reports twice a year. Our aim is to ask for reports four times a year.
• We lack data on the films that are not supported by Telefilm Canada. The only source of information is the assessments by the producers, depending on when and if they have the time and/or interest to provide us with it.

4. Success evaluation methodology

• Scores are calculated for individual films, production companies and creative teams.
• The index that an individual film gets is a proxy, and not the real value of a specific box office.
  • We have a special index calculation method for smaller art films in order to make them more competitive, and because Telefilm Canada supports mainly that sort of films.
  • Films with a 5M CAD box office get the same index as films with a 40M CAD box office, to ensure fairer distribution of our support.
  • The scales have become similar for domestic and international film sales.
  • There is also a new scale for selections and awards at international and domestic festivals (see the enclosed presentation for more details).
• Each film gives a score to their production company.
  • We calculate each company’s average index based on the cumulative results of all the films produced over a five-year period.
  • This average index serves as a tool for assessing and predicting the success and feasibility of each company’s future applications.
  • Scoring is gradually replacing CVs.
  • It provides a clue as to whether a production company will be able to deliver what it has promised.
• Creative teams are also assessed.
  • The director, for example, gets his index for every single film he has directed, and an average index for a five-year period.
  • The same applies to the producer and the scriptwriter.
  • The producer usually gets the same index as the production company since, mostly, he runs it himself. In the case of a company hiring a producer for but a single project, a different index is made available.

5. How does the index impact the decision-making process?

• At the project level, these are the five decision-making criteria:
  • the production company’s track-record as shown by the company’s index;
  • the creative team members’ track record as shown – but not exclusively – by the creative team’s indexes derived from their filmography;
  • the relative market interest expressed by the level of financial support the project earns;
  • the marketing approach and strategy, especially the initial promotional push;
  • the quality of both the script and the director’s vision.
• We check the market interest for every project, and engage in a lot of discussions with distributors in this respect:
  • For budgets under 2,5M CAD, the producer is not required to demonstrate market interest when applying, but such projects never get over 500,000 CAD funding.
  • For budgets over 2,5M CAD, the producer must demonstrate 10% market interest. Market interest could be expressed through a fan-base, kick-starter mechanism, national and international MGs, gap financing, sponsorships, etc.
  • Sometimes we use our index to indicate that even for projects below 2,5M CAD, there may be market interest but that does not influence the final decision whatsoever.
• We also take into consideration the global portfolio of our investment, based on diversity of genres, projects coming from the different provinces of Canada, minority groups — e.g. Aboriginal communities, the Francophone community outside Quebec, the Anglophone community in Quebec—and new talents. Around 10% of the budget is given to new talents in the French-speaking fund, and around 15% in the English-speaking fund.
• We have also removed deadlines. We are spending more time on discussions and conversation with producers about the ideal timing for submitting their application – when they have a cast in place or the proof of market interest, etc. We try to be fair, and we tell producers not to bother to apply if they have not met certain criteria. Deadline removal works well, especially within the English-speaking market, because it fits in better with the mentality of producers working there.
• We replaced our automatic system with a fast-track system, designed for successful companies that produce at least three films in five years. Five companies have been selected and have access to reserved and additional funding.

Challenges

• The index is not strong enough to ensure what will be successful. It is based only on a company’s previous score. During the decision-making process, we still need CVs and the track records of directors, scriptwriters and individual producers.
• We still do not know how to fully define a sustainable company. How many productions does it have to do per year? What kind of revenue to generate, etc?
• Even if this system is innovative, it is just information. At the end of the day, it is still a a human being who makes the final decision.

6. Future plans

• To increase the frequency of the revenues calculation from twice a year to four times a year.
• For films supported by Telefilm, to track the number of VOD transactions and other kinds of online distribution access in order to include the results in the calculation of the indexes.

• To check audience demand, to harmonize and standardize measurements at a consumer-centric level. At the moment, we only consider the the supply side.
• To better measure the opinions of consumers/critics.
• To further automate data collection and calculations.
• To improve our ability to adapt the scheme to the changes in the cultural sector, in order to remain relevant
• To further develop international collaborative efforts to build and maintain modern measurement frameworks. The Success Index has been presented to film funds in other countries to obtain feedback, and to let them understand that films should not be measured only according to the US-imposed box-office standards.

Outcomes of group discussions

• Such an index can make funds both player and pawn, depending on what they need. Funds can play a pawn and reject a project on the basis of its poor index. However, if they really like a project, they can be a player, and decide to support it in spite of the poor index.
• The indexes show what has been achieved as a fund far better than any other figures.
• This system pushes production companies forward; it encourages them to become proactive and to have a vision at all times, knowing that they are under constant evaluation.
• It is great that such a tool includes more than merely economic value, but also the cultural aspects of projects.
• Today, funds need tools to justify their decisions and expenditures because the number of applications and films is on the increase daily. They need a system that is easy to apply but sophisticated in terms of the result it conveys, and what it defines as success.

Challenges

• This is an objective tool and provides good arguments to evaluate production companies using the same criteria for all. Nonetheless, it cannot be applicable to every market. For a small market with only small companies that produce one film every three years or more, and where there are almost no changes over years, this instrument may prove too costly to develop.
• The context should be taken into consideration when reporting why a project has a bad score. In the Netherlands, for example, 50,000 viewers for a children’s film is reported as a high score. Yet in comparison to average results, such a score is not high at all. Or else a film could have poor results simply because the competition was particularly high when it was released. Automatized success evaluation tools can endanger this approach.

Promotion, Distribution and Success Evaluation

• Module 1 – The Role of Public Funds in Promotion
• Module 2 – Distribution – Who and What to Support
• Module 3 – Vod Platforms as Potential Friends of the EU Cinema
• Module 4 – Release Windows
• Module 5 – Finding and Addressing Audiences
• Module 6 – How to Evaluate Success

• List of Participants (PDF)
• MEDICI Fourth Workshop Full Report (PDF)

Illustrations by Mišo Duha

Schedules Previous Workshops Partners Contact