We are pleased to announce that we are now officially on Linkedin to engage and connect with our clients and people interested in digitalisation, data science and technology for management and evaluation of international development and humanitarian actions. Make sure you check our company page regularly, as we will update you on current projects and any potential job opportunities. Connect with us and join the exchange! #projectmanagement #datascience #internationaldevelopment

Reproducible samples and analyses are critical for data quality, particularly in monitoring and evaluation of project activities. Okay, some may say, but does it really matter at all? Yes, it does. It helps setting a seed for the future.

Evolution in project monitoring and evaluation

Let us take some capacity-development activity such as a training as an example. In the past, it might have been easy to simply write a “qualitative” description of the contents of a training with some technical jargon / acronyms and mention the number of participants that were there (hopefully also being able to provide or “estimate” a percentage of women among participants).

Monitoring and reporting project implementation was largely seen as a bureaucratic requirement. There was no structured way to learn from participants. There was virtually no systematic data collection for knowing how to improve training contents, methods, materials and results based on participants’ views and suggestions (feedback).

This fitted well with the traditional top-down approach to capacity development. In a way, it also made sense. Digital data-collection tools were complicated to use and considered expensive to maintain. They also represented more pressure on project managers and staff.

Project managers could be confident (and I am afraid some still are) that treating reports merely as regular “literary” exercises while focusing efforts on financial compliance would be enough. After all: “in the end we will write something nice in the implementation report”. Learning from project implementation and evolving from experiences were suffocated by a binary logic of “success” or “failure”. In such a context, it is easy to miss the fact that experimentation “failures” are important steps towards learning for impact success.

Paradigm change

This context has been changing fast in the era of data abundance and analytics. Many still see terms such as “automation” and “machine learning” as threatening. Personally I think that improvised, unstructured and scientifically-weak monitoring, evaluation, accountability and learning systems have done enough harm in terms of loss of resources and opportunities. This is particularly so in the public and international development sector. It is great to see that things are finally evolving from discourse to practice.

Learning from experience is gradually becoming easier and cheaper. The powerful and open source computational tools that are freely available such as R and Python can make it easier to reduce sample-selection bias but require at least some basic knowledge of their syntax. Many organisations are still adapting to the paradigm shift from top-down, specialised expertise to a more collaborative and multidisciplinary data-driven approach to monitoring, evaluation and learning. This process requires data-science skills that blend computing and statistics following professional monitoring and evaluation standards. Investment in human resources and targeted recruitment / contracting are key. Data management and analysis using traditional spreadsheet software such as MS-Excel and conventional, proprietary statistical packages (e.g., SPSS and STATA) are not enough anymore for a world with complex, unstructured data.

Sampling in a scientifically-robust (but simple) way

A common question that clients have asked me is about how best to select participants for feedback surveys in activities such as training and events. Thinking about this and the context above I developed a very simple app using Shinyapps. This app generates a reproducible random sample list of numbers. Samples are reproducible with the function set.seed(44343) using R.

You can access and use the app at https://movimentar.shinyapps.io/randomsample/. You will simply need to input the total number of participants in the event/activity, and a sample percentage. This will depend on the size of your activity. After that, you can visualise the result in a reactive table and download the output in XLSX format (MS Excel).

The “magic” here is that if anyone executes set.seed() with the same number specified between the parentheses, one will always see the same randomised sample. This makes it reproducible while avoiding the problem of sample-selection bias. So, people in the future can also learn from your experience with assurance that you put some thought into data quality.

It is also possible to draw reproducible samples in many other statistical computing languages. In Python, for example, you should import random and call random.seed() to set your seed number. After that you need to import numpy and call numpy.random.choice() to get your sample. However, be aware that the seed number (44343) used as reference in the randomsample app will generate a different sample in Python as the app is built in R.

The app’s source code is publicly available for download on Github. I hope that this helps others to learn more about these tools. Code contributions will be very welcome too.

Let us learn for real. It is time to set.seed() for the future.

Written by: Eduardo W. Ferreira, PhD / Consultant, data scientist, trainer and facilitator. Eduardo supports designing, managing and evaluating projects and programmes for consultancy firms, non-governmental organisations, governments, research institutions and international organisations (Additional information).

Project management has some things in common with playing with a kite. One needs to adapt well and quickly to changes in external conditions following observation of performance. Otherwise, one runs the risk of blindly hitting the ground.

Differently from kites, though, a project or a programme requires more than open eyes. It requires sound data collection and analysis. Beneficiary / client feedback in opinion surveys can help tracking performance, user satisfaction and improvement needs. This is particularly so if one uses reproducible, computer-based random samples in line with professional statistical/data-science standards. This is the way to learn from project implementation based on modern scientific and computational methods.

Lesson learned

Some time ago, I had a consulting assignment with a youth-violence reduction project in Brazil. The project needed baseline data for its logframe indicators. So, we designed a system for collecting, storing, processing and reporting data using Open Data Kit in Android devices. We also used R (statistical-computing language) for programming a reproducible sample, as well as all data processing and analytical reporting.

Before collecting data, I had to train over 25 people including interviewers and partner staff, who also suggested changes to the data collection form. This post is about one of these suggestions, which was a particularly good lesson learned.

Jane Davidson’s article “Breaking out of the Likert scale trap” inspired me to propose the inclusion of direct evaluative questions instead of the traditional Likert scales. It is a very good post claiming that by using evaluative terms right in the questionnaire, participant ratings become a lot easier to interpret in terms of quality or value. I also think so.

The Likert scale using “strongly agree” to “strongly disagree” is great for assessing opinions and knowledge from respondents. However, the scale makes it difficult to draw evaluative conclusions on quality or value of a training workshop, project or programme, for example. So, the scale suggested by Davidson was as follows:

1. “poor / inadequate”;
2. “barely adequate”;
3. “good”
4. “very good”
5. “excellent”

The draft data-collection form used the same label categories as those above but translated to Portuguese. During the interviewer training workshop, one of the participants spotted a potential problem that I also did not notice before. The label categories were not well balanced…

The problem was that in the scale above there are three positive and two negative scale categories or levels. Hence, the likelihood / probability of a positive result tends to be higher. Those unbalanced options are a potential source of bias.

For preventing such bias, we changed the labels proposed by Davidson to:

1. “very poor” or “very low”
2. “poor” or “low”
3. “regular” or “average”
4. “good” or “high”
5. “very good” or “very high”.

Additional answer categories

I would recommend to include the categories “Not sure, I don’t know”, “Not applicable”, in order to allow a more complete respondent feedback. The numeric scale can integrate these new categories depending on the question (e.g., answering not applicable or reporting not to know the action under evaluation can also indicate the quality of its outreach and impact).

Sometimes, it can also be interesting to have the answer option “I do not want to answer” for sensitive questions about income or abuse, for example. This option, of course, should not be part of the numeric evaluative scale. Otherwise, one will mix up different types of result.

Numeric analysis

The corresponding numeric intervals must also be balanced.

For a scale from 1 to 5 (one being the worst case, as in the article from Davidson, or the other way round as it is the case in Germany where the score one is the best), the interval from the function “cut” in R (statistical computing language) is:

> cut(1:5, breaks = 5)

[1] (0.996,1.8] (1.8,2.6]   (2.6,3.4]   (3.4,4.2]   (4.2,5]

This would be equivalent to:

• from 1 to 1.80: “very poor/very low”
• from 1.81 to 2.60: “poor/low”
• from 2.61 to 3.40: “regular”
• from 3.41 to 4.20: “good”
• from 4.21 to 5.00: “very good”

The same can be done for a scale based one the interval from 1 to 7 if one includes the categories “Not sure, I don’t know”, “Not applicable”. The R output from the cut function for a scale with seven categories is as follows:

> cut(1:7, breaks = 7)

[1] (0.994,1.86] (1.86,2.71]  (2.71,3.57]  (3.57,4.43]  (4.43,5.29]

[2] (5.29,6.14]  (6.14,7.01]

Preventing response bias

For further preventing bias, the survey introduction can try to make survey participants aware about the risk of providing biased answers. An introduction following the paragraph below can help:

Respondents in such questionnaires sometimes repeat the same answers for different questions, mark extreme answers trying to be polite or as form of calling attention to a specific aspect, or even rate items in the middle categories in order to keep neutrality when they are actually thinking something else. Please avoid this as much as you can, as it prevents us from understanding the real situation.

If you are asking for real feedback from clients/beneficiaries and stakeholders, interviewers must be external to your project team. Ideally, they should be outsourced and receive training on interviewing methods and not associated to the implementing organisations or related to their staff members. This helps preventing interviewer bias (when results are different depeding on who collects data). This can be the case, for example, when humanitarian-aid beneficiaries have suggestions for support improvement but fear loosing future support after having provided critical feedback.

Final remarks

I benefited from Davidson’s contribution and I thought it would be good to try to contribute as well. Monitoring and evaluations with robust scientific standards can powerful for learning and improving policies, programmes, projects and products.

The evaluative scales can be very helpful but it does not mean that Likert scales should be avoided by all means. I also use Likert scales in my forms, particularly in those aiming to test subject knowledge from participants in capacity development actions such as projects including training workshops or a course module.

Also, it is worth including an open question about problems (e.g., What are the three main problems in your village?) as well as an open question about suggestions for improvement or additional comments. Text data can be analysed with word clouds and dendrograms, for example. This can complement well scoring data in monitoring and evaluation. It is also an opportunity for projects and programmes to track opportunities while making sure that they are addressing the issues that their beneficiaries or clients consider most important.

I hope you enjoyed this post and would be happy to receive any suggestion or comment.

Good monitoring and evaluation! 

In this post I try to explore this and some other questions using open-source statistical computing R language and public recruitment data from CharityJob’s website. According to CharityJob, the site is the United Kingdom’s busiest one for charity, fundraising, NGO and not for profit jobs.

In addition to presenting these powerful open-source tools and data-exploring techniques, I hope that this post can help the public, specially applicants and workers to get an update on salaries and trends in the sector. The jobs analysed here are mostly UK-based ones and published by UK-based organisations. Therefore, the results below are not meant to represent the entire sector worldwide. I still hope though that this post can provide some positive contribuition to the evolution of the sector in both the southern and the northern hemispheres.

For those of you who are only interested in the end analysis, please jump to the results section. However, I encourage you to explore how these tools work. I believe that they can help speeding up and improving quality of the so-much-needed charity, social-enterprise, development-aid and humanitarian work globally.

I used here some basic techniques of web scraping (web harvesting or web data extraction), which is a computer software technique of extracting information from websites. The source code in RMarkdown is available for download and use based on GNU General Public License at this link: Rmarkdown code. Everything was preapred with the open-source, freely-accesible and powerful statistical computing language “R” (R version 3.2.0 ) and the development interface RStudio (Version 0.99.441).

This post is based on public data. The post is my sole responsibility and can in no way be taken to reflect the views of CharityJobs’ staff.

Downloading data from CharityJobs

Using RStudio, the first step is to download the website data. CharityJobs’ search engine contains over 140 webpages, each of them with a list of 18 jobs in most cases. Hence I expected to get information about around 2,500 job announcements. For that, the first step was to download the data and get rid of what I did not wanted (e.g. css and hmtl codes). The code chunck below describes how I did it. The code contains explanatory comments indicated by hashtags (‘#’). I am sure that many would be able to write this code in a much more elegant and efficient way. I would be very thankful to receive your comments and suggestions!

#  Loading the necessary packages. It assumes that they are installed.

#   Please type ‘?install.packages()’ on the R console for additional information.  

    suppressWarnings(suppressPackageStartupMessages(require(rvest))) # Credits to Hadley Wickham (2016)

    suppressPackageStartupMessages(require(stringr)) # Credits to Hadley Wickham (2015)

    suppressPackageStartupMessages(require(lubridate)) # Credits to Garrett Grolemund, Hadley Wickham (2011)

    suppressPackageStartupMessages(require(dplyr)) # Credits to Hadley Wickham and Romain Francois (2015)

    suppressPackageStartupMessages(require(xml2)) # Credits to Hadley Wickham (2015)

    suppressPackageStartupMessages(require(pander)) # Credits to Gergely Daróczi and Roman Tsegelskyi (2015)

    suppressPackageStartupMessages(require(ggplot2)) # Credits to Hadley Wickham (2009)

    ## Creating list of URLs (webpages)

      urls  <- paste(“https://www.charityjob.co.uk/jobs?page=”, seq(1:140), sep = “”)

  ## Downloading website information into a list called `charityjobs` and closing connections

      charityjobs  <- lapply(urls, . %>% read_html(.))

Tyding up and parsing data

The next step is to parse or clean up the text string of each of the about 140 webpages. I decided to build a custom function for that, which I could use to loop through the content of each element of the charityjobs list. The function should also save the parsed data into a data frame. This data frame should include information on recruiters, position titles, salary ranges and deadline data. The code chuck below presents this function, which I called salarydata.

## Creating a function for parsing data which uses the read_html output (list ‘charityjobs’)

salarydata <- function(list) {

  # Creating auxiliary variables and databases

  list_size <- length(list)

  salaries  <- data.frame(deadline=character(),

                          recruiter=character(), 

                          position=character(),

                          salary_range=character()) 

  for (i in seq_along(1:list_size)){ 

    size <- list[[i]] %>% html_nodes(“.salary”) %>% html_text() %>% length()

    #Intermediary dataframe

    sal  <- data.frame(deadline=rep(NA, size),

                       recruiter=rep(NA, size), 

                       position=rep(NA, size),

                       salary_range=rep(NA, size)) 

    ## Filling out intermediary data for deadlines for application

    sal$deadline[1:size]  <- list[[i]] %>% 

      html_nodes(“.closing:nth-child(4) span”) %>% html_text() %>% 

      .[!grepl(“^[Closing:](*)”,.)]  %>% rbind()

    ## Filling out intermediary data for recruiters

    sal$recruiter[1:size]  <- (list[[i]] %>% 

                                 html_nodes(“.recruiter”) %>% html_text() %>% 

                                 gsub(“\r\n\ \\s+”, “”,.) %>% 

                                 gsub(“\r\n”, ” “, .) %>% 

                                 gsub(“^\\s+|\\s+$”, “”, .)) %>% 

      rbind()

    ## Filling out intermediary data for positions

    sal$position[1:size]  <-   list[[i]] %>% 

      html_nodes(“.title”) %>% html_text() %>% 

      gsub(“\r\n\ \\s+”, “”,.) %>% 

      gsub(“\r\n”, ” “, .) %>% 

      gsub(“^\\s+|\\s+$”, “”, .) %>% 

      rbind()

    ## Filling out intermediary data for salary ranges

    sal$salary_range[1:size]  <- list[[i]] %>%

      html_nodes(“.salary”) %>% html_text() %>%

      gsub(“(£..)\\.”, “\\1”, .) %>% gsub(“\\.(.)k(+) |\\.(.)K(+)”, “\\100 \\2”, .) %>%

      gsub(“(*)k(+) |(*)K(+)”, “\\1000 \\2”, .) %>%

      gsub(“k”, “000 “, .) %>% # Substituting remaining ks

      gsub(“^(£..)\\-“, “\\1000; “, .) %>% # Adding thousands for figures withou “k”

      gsub(“- £”, “; “, .) %>% gsub(“-£”, “; “, .) %>% gsub(“£”, “”, .) %>% # Removing pounds signs

      gsub(“-“, “;”, .) %>% gsub(“–”, “;”, .) #%>% # Removing dashes

    ## Excluding per-hour and per-day jobs

    sal <-  sal %>% filter(!grepl(“hours”, sal$salary_range))

    sal <-  sal %>% filter(!grepl(“hour”, sal$salary_range))

    sal <-  sal %>% filter(!grepl(“p/h”, sal$salary_range))

    sal <-  sal %>% filter(!grepl(“week”, sal$salary_range))

    sal <-  sal %>% filter(!grepl(“ph”, sal$salary_range))

    sal <-  sal %>% filter(!grepl(“day”, sal$salary_range))

    sal <-  sal %>% filter(!grepl(“daily”, sal$salary_range))

    sal <-  sal %>% filter(!grepl(“plus”, sal$salary_range))

    sal <-  sal %>% filter(!grepl(“\\+”, sal$salary_range))

    salaries <- rbind(salaries, sal) 

  }

  return(salaries)

}

Creating full dataframe and other adjustments

The last step before exploring the data was to run the function salarydata to create the full dataframe. After that, I parsed lower and upper salaries into separated columns, deleted data which may have been incorrectly parsed or data concerning daily-rate and hourly-rate jobs / consulting assignments. Only yearly salaries between GBP 4,000 and GBP 150,000 have been considered. All salary data is in British Pounds (GBP) and refer to annual salaries, which sometimes do not include benefits such as pension.

Cleaning the salary-range variable was a tricky step as the website allows users to type in both salary amounts and additional text (e.g. 30,000, 30K, or 25-30k). Therefore, I had to iterate some times until the output was good enough. I am quite sure that the code chunk below can be written in a more elegant way. Again, please let me know in case you have any suggestions here.

# Creating a full and clean dataframe

  salaries <- salarydata(charityjobs)

# Parsing salary-range variable

  salaries$salary_range <- gsub(“, “, “,”, salaries$salary_range) %>% 

    gsub(” ; “, “;”, .) %>% gsub(“; “, “;”, .) %>% 

    gsub(“,[:A-z:]”, ” “, .) %>%

    gsub(“\\(*”, “”, .) %>% 

    gsub(“\\:”, “”, .) %>% 

    gsub(“[:A-z:],[:A-z:]”, ” “, .) %>% 

    gsub(“(..),00\\…”, “\\1,000”, .) %>% 

    gsub(“(..),0\\…”, “\\1,000”, .) %>% 

    gsub(“[A-z]”, “”, .) %>% gsub(“,”, “”, .) %>% 

    gsub(“\\.”, “”, .) %>% gsub(“^\\s+”, “”, .) %>% 

    gsub(“\\s([[:digit:]])”, “;\\1”, .) %>% 

    gsub(“\\s+”, “”, .) %>% gsub(“^[[:digit:]]{1};”, “”, .) %>% 

    gsub(“\\(“, “”, .) %>% gsub(“\\)”, “”, .) %>% # Deleting “(” and “)” 

    gsub(“\\/”, “”, .) %>% gsub(“000000”, “0000”, .) %>% # Deleting “/” and correcting digits

    gsub(“([[:digit:]]{2})00000”, “\\1000”, .)  %>%  # Correcting number of digits

    gsub(“([[:digit:]]{5})00”, “\\1”, .) # Correcting number of digits 

# Adjusting data and computing lower and upper salaries using “;” as separator

  salaries  <- suppressWarnings(salaries %>% 

  mutate(upper_salary=gsub(“^.*;”, “”, salaries$salary_range)) %>% 

  mutate(lower_salary=gsub(“;.*”, “”, salaries$salary_range)) %>% 

  mutate(upper_salary=as.numeric(upper_salary)) %>% 

  mutate(lower_salary=as.numeric(lower_salary)) %>% 

  filter(upper_salary<150000) %>% filter(upper_salary>4000) %>%

  filter(lower_salary<150000) %>% filter(lower_salary>4000)  %>%

  mutate(lower_salary=ifelse(lower_salary>=upper_salary, NA, lower_salary)) %>%

  filter(is.na(upper_salary)!=TRUE) %>% tbl_df() %>% 

  select(deadline, recruiter, position, 

         lower_salary, upper_salary, salary_range) %>% 

  mutate(deadline=dmy(deadline)))   

The output below presents the summary of the full dataframe (first 10 observations).

## Source: local data frame [1,704 x 6]

## 

##      deadline                 recruiter

##        (time)                     (chr)

## 1  2016-09-04                       ZSL

## 2  2016-09-12 Alliance Publishing Trust

## 3  2016-08-31         Save the Children

## 4  2016-08-30         Blind Veterans UK

## 5  2016-09-08            Headway SELNWK

## 6  2016-08-30                Saferworld

## 7  2016-09-22               Pro-Finance

## 8  2016-09-06           TPP Recruitment

## 9  2016-09-06               Harris Hill

## 10 2016-09-20    Hays London Ebury Gate

## ..        …                       …

## Variables not shown: position (chr), lower_salary (dbl), upper_salary

##   (dbl), salary_range (chr)

Results

The final dataset contains information of 1,704 jobs of various types, based on yearly-salary figures. They exclude consultancy assignments and other jobs based on hour and day rates as well as jobs which did not provide salary information. The table below presents the summary statistics concerning the lower and upper salaries.

The table below presents standard descriptive statistics for lower and upper salaries. For job announcements providing a single value (not a salary range), that single amount has been incorporated to the dataset variable upper_salary while the variable lower_salary was set as NA (not available). That is why the number of observations (N) is 785 for lower salaries and 1,704 for upper salaries. About 54% of the job announcements did not provide salary range information but just the single salary amount.

Summary statistics of salaries (in British pounds / GBP)

In a more in-depth analysis for some future post, it can be interesting to look into payments for jobs paying by hour and by day as well for more specific job categories. One way for approaching specific job categories can be by defining tags for job titles using standard words from titles (e.g., director, management, assistant) and groupping them by tag type in a new factor variable.

Histogram with distribution of lower salaries (GBP)

Histogram with distribution of upper salaries (GBP)

The 10 most frequent recruiters

The table below presents the ranking of the 10 most frequent recruiters in the dataset. Column “N” presents the number of total announcements for each recruiter while column “Freq” shows the percentage of total announcements for each recruiter. Among these are also recruitment agencies.

The tables below show the ranking of the jobs with the 10 lowest and 10 highest upper salaries. 

The jobs with the 10 lowest upper salaries (GBP)

The jobs with the 10 highest upper salaries (GBP)

I also wanted to quickly explore possible relationships between deadline dates and salary levels, just for fun. It could be, for example, that some periods had lower average-salary offers than others.

Despite the large number of job announcements in the dataset (N=1704), all observations refer to jobs with application deadlines between 24 August 2016 and 23 September 2016. This is a short time span for such analysis, but I explored it anyway just as an example of what these tools and techniques can do.

The plot below shows the mean (or average) upper salary for each day throughout the period. The variation in the mean salary as well as salary levels seem higher for jobs with deadlines as from September. The dashed line represents the results of the linear regression. The linear model however fails to detect any statistically significant relationship between mean salary and application date (R² = 0.006; p = 0.69).

Average upper salary by date (GBP)

Next, I will use word clouds to explore job titles. The larger the word in the cloud, the higher is its frequency in the dataset. The words below are only those mentioned in at least 10 job announcements. The plot indicates that management positions are the most frequent ones, followed by coordination jobs, as well as officer, recruitment and fund-raising jobs.

Word cloud of job titles

The cloud below shows the most frequent words in the names of the recruiting institutions. I assumed that its results could provide hints about the most active thematic areas in terms of job anouncements. The words in the plot below are also those which have been mentioned in at least 10 job anouncements. The word cloud suggests that recruitment agencies are among the leading ones, as expected (see section “The 10 most frequent recruiters”). Organisations working with children, cancer and alzheimer patients also seem to stand out.

Word cloud of recruiters

Moving forward

The charity, development aid, not-for-profit and social enterprise sector is evolving rapidly. This process is powered both by increasingly-critical global challenges and, of course, by capable and motivated entrepreneurs, staff and service suppliers. This is a sector which is sometimes too much romantised by some people. As a consultant and entrepreneur in the sector, I am often asked how I manage to deal with all day dreamers I come accross in my way. No judgment about that but this indicates how much the sector is still unknown to the public. This is a sector which has become increasingly professional and results oriented. I believe that computing for data analysis can help the sector, particularly concerning monitoring and evaluating performance, which should include staff and beneficiary / client satisfaction.

I hope you enjoyed this tour and would be happy to receive your suggestions for additonal analysis and improvement. You can access this post with more updated data at: https://rpubs.com/EduardoWF/charityjobs.

Keep coding and take care!

Written by: Eduardo W. Ferreira, PhD / Consultant, data scientist, trainer and facilitator. Eduardo supports designing, managing and evaluating projects and programmes for consultancy firms, non-governmental organisations, governments, research institutions and international organisations (Additional information).