cft

The most important thing for Udemy could be to solve for course discovery & recommendation

Udemy is a marketplace platform for learners and instructors


user

Abhishek Rai

3 years ago | 8 min read

Problem #1: Ineffective Course Discovery

Udemy is a marketplace platform for learners and instructors. One of the core jobs of the platform is to enable fast & relevant product discovery. Currently, customers face problems in selecting the right course on Udemy and similar UpSkilling Platforms (excluding platforms offering degree programs).

The company seems to rely on the course catalog, the number of enrolments, and learner reviews for facilitating discovery. There are other metrics such as course duration and course language as well.

But these do not seem sufficient which we will explore in detail through this article. Below is a quick example of two seemingly similar courses (ratings, numbers of enrolments, course descriptions are similar): which course should a learner choose?

Example 1 — Excel course with 18 hrs duration

Example 2 — Excel course with 27hrs duration

Impact: Inefficient discovery results in sub-optimal RoI of time & money invested by learners and the same can be quantified through the following proxies:

  • More time spent on selecting a course
  • Low course completion rates (or high drop out rates) due to not able to meet learner’s expectations

Problem #2: Ineffective Course Recommendations

Below is an example that highlights challenges with the current recommendation system. Out of the top 5 courses shown to the person who did a course on value investing, 2 are on learning Spanish.

One would expect either accounting, corporate finance, business moats, capital allocation, or other skills that help in understanding the core subject (in this case ‘value investing’) better.

Example: Current Recommendation at Udemy

Impact: Poor recommendations are likely to fail in persuading learners to do more courses thereby resulting in low conversion through upsell/cross-sell nudges, and reducing LTV from learners.

I think the problem of search and recommendations are related in the case of Udemy. But before we delve into it, let’s talk about learners and how do they approach upskilling:

  • The need for learning or advancing a skill is driven by some use-cases. It can be either performance at work, hobby, start-up, interview prep, etc.
  • Learners have different starting points (in terms of prior exposure), varying capabilities/IQ, and expectations related to learning outcomes depending upon their use case. One can say that learners are highly fragmented on attributes (prior knowledge, IQ, desired outcomes) that are difficult to capture.

One may also argue that B2C markets are always fragmented e.g. e-commerce so many different types of users searching for so many product categories, varieties, brands, etc.

But in e-commerce, products & benefits are tangible. Reducing user and product fit can be thought in two stages: (a) capturing user’s requirements (b) mapping product features with user’s requirements.

E.g. while purchasing a Air Conditioner user’s requirements could be AC for 1 BHK, 2 BHK. On product feature side, AC tonnage relates with approx. house area.

Now to reduce the user-product fit, two approaches can be taken: either user becomes aware of product features (in this case right tonnage) or features are translated into user’s requirements (e.g. adding product filter like Buying for 1BHK, 2BHK).

Companies attempt to educate customers on product features by creating buying guides.
The point is, if product benefits are tangible then mapping user requirements with product features become easier. In Udemy’s case, product features are not standardized nor user requirements and hence managing learner-course fit becomes challenging.

  • Learners are subconsciously evaluating options so as to optimize their RoI, where they are investing time & money while on return side they are looking to acquire relevant skills for their use-cases.

Now where does the problem lie:

  1. No standard ways of comparing overwhelming numbers of options within & outside Udemy. E.g. you can compare two laptops or washing machines on standard configurations. How do you compare any two courses?
  2. Learner-course misfit: One size does not fit all
    Supply-side: (a) In an attempt to attract a large number of learners, many instructors tend to create broad courses with universal appeal e.g. one is likely to find more courses on themes such as ‘Excel for anyone’ or ‘Excel for beginners’ instead of ‘Excel for sales team’ or, ‘Excel for engineering undergraduate’’.

(b) Variability in instructors’ capabilities that results in varying quality of courses (Udemy does not verify knowledge or experience of instructors)

  1. Demand-side: We already discussed there is no standard way to capture demand-side (need) fragmentation. E.g. an excel course may be too basic for a few and too advanced for others. What is basic and advanced may vary for different learners. Similarly, the pace of the course may be slow/fast.

Current solutions:

For “No standard ways of comparing options”, solution is missing; learners need to go through the various syllabi and compare them. There is no solution that compares and contrasts courses on standard parameters.

To solve for the “Learner-course misfit, there are four instruments which are used:

  • Course descriptions: A detailed description of the course deliverables to set the right course expectations for learners. The expectations are that learners should be able to make a decision after reading the course description.
  • Learner reviews: Feedback from past learners to determine course & instructor quality.
  • A lot of focus on beginners and getting folks started e.g. adding tabs like ‘beginner favorites’.
  • Learner-friendly refund policy as risk-cover i.e., learners can request a refund up to 30 days after enrolling in the course — eliminates monetary risks for learners.

Udemy’s Money Back Guarantee

It is assumed that details of course deliverables along with student reviews can help learners in selecting the right course. Even after this if the decision goes wrong, one can request for refund.

Ineffectiveness of current solutions:

  1. Course descriptions:
    (a) These are created by instructors and tend to have persuasive language, often overestimating deliverables.
    (b) Detail listing of syllabus helps only the ‘most aware’ learners. It can be assumed that ‘Most aware’ learners are likely to figure out what they need to learn. But many learners are not in that category and they just know their use-cases & pain points but don’t know what they should be learning to solve for them.
  2. So mentioning the detailed syllabus does not solve the real problem.
    Example: For a marketing team, instead of mentioning course covers lookup, index(match), etc. it shall mention that one will be able to analyze lead flow, campaign RoI, budget allocation & tracking, etc.
  3. Learner reviews:
    It does not help in situations when two courses have seemingly equal ratings but the different focus, duration e.g. both ‘excel for beginners’ and ‘excel from beginner to advanced’ may have 4+ ratings but course duration may differ e.g. 4 hrs vs 20 hrs.
  4. Focus on beginners:
    It works! A person with no idea about the field is going to like the course as long as the instructor is great. But:
    (a)It also has the potential for sub-optimal results as learners would not know what would have been best for them. A simple way to understand this could be that a learner is brought from zero to 50% but did she/he also have options which could have brought them at 80% and she/he could not search them?
    (b) Further, it does not help with a good learning path. After having done a good beginner course, how do I choose an advanced course?
  5. Refund policy:
    It comes at the cost of customer experience. Many courses on Udemy are already at a low price point hence monetary risk for learners is not high. The refund process results in significant time waste e.g. lost time in purchasing, going through an ineffective course, and then requesting a refund. How does one ensure the best RoI on time invested?

Apart from this, as already discussed the solution for the course comparison piece is completely missing.

Examples from other industries:

The product discovery problem is not unique to Udemy. Any marketplace where users need to search & select is likely to face similar issues e.g.

  1. Online Streaming Platforms: Ever-increasing content library, need to personalize for users. Netflix is probably leading the efforts on search, personalization space
  2. E-commerce: Product search optimization & recommendations are topics of great interest among e-commerce firms. Check this note by Flipkart

How can Udemy solve for this:

The end-goal is to help learners select the right course. It can be broadly be split into sub-parts such as:

  • Show relevant search results
  • Help them compare courses effectively
  • Show better recommendations to complete their learning journey

Following actions can help achieve these:

(1) Start capturing user persona and show search results and recommendations based on persona: People in a similar age group pursuing similar paths are likely to have similar upskilling requirements as their end use-cases are likely to be similar. For implementation purposes, Udemy can start capturing user persona details and what they are learning. E.g. a persona could be ‘engineering undergraduate’ (solves for search, recommendation)

(2) Educate learners: Creating a simple guide on ‘what to learn & how to select courses’ for the top fields. I expect that Pareto rule will be applicable here, the top 10% fields would be attracting 80%+ enrolment. Creating such guides for even these top 10% of fields shall help a large number of learners. E.g. a guide could be ‘What should you learn in Excel?’ (solves for compare)

(3) Lead the way — define a blueprint of ‘proficiency stages’ for different skills, tag existing & new courses, use these tags to compare & contrast, arrange these tags to create learning paths

What does proficiency stages mean? Udemy can create learning tracks e.g. Excel Stage 1, Stage 2, Stage 3 courses, and map specific items under each of the stages. Each stage shall have specific learning goals; Udemy can also create independent hurdles/assessments for each of these stages.

Implementation? The massive task would be to get all the educators to map their courses as per these stages. Instructors will have options to either create dedicated courses for Udemy’s track or their own independent tracks. Irrespective of that, they shall map the curriculum of the course in the Udemy’s pre-defined item list.

E.g. an instructor may create a course but may not cover all the items of Excel Stage 1 and 2 but maybe somewhere in between.

What capability does the platform gain? Such tagging will help facilitate the creation of a similarity matrix that can help learners see what is common and what differs between two courses. Such proficiency stages can also be used to create/recommend a learning path for users to upskill further.

How will it benefit learners? Learners will have a tangible idea about their skill level. They can more objectively compare and select courses and, decide the kind of depth they want to pursue.

How will it benefit Udemy? Udemy as a platform will be able to facilitate better & faster search based on proficiency and, recommend an appropriate learning path for learners.

It will improve the overall learner experience and result in an increase in the LTV. (solves for search, compare, recommendation)

An advanced version of ‘proficiency stages’ & learning paths could be to adjust these based on the persona & regularly update it as more user data is gathered.

Endnote:

I understand that this article does ‘arm-chair’ thinking and, each of these ideas would need to be validated with a lot of data that Udemy may already have; further, hypotheses would need to be defined precisely and translated into smaller experiments to be able to select the best way forward.

But looking at current flow, I strongly feel that course discovery needs to be improved significantly, and solving for it can improve RoI on invested-time for learners and may make Udemy more valuable as a platform.

Would love to get your thoughts on this.

This article was originally published by Abhishek rai on medium.

Upvote


user
Created by

Abhishek Rai


people
Post

Upvote

Downvote

Comment

Bookmark

Share


Related Articles