{"id":20005,"date":"2017-06-14T08:51:49","date_gmt":"2017-06-14T12:51:49","guid":{"rendered":"https:\/\/carleton.ca\/edc\/?p=20005"},"modified":"2021-08-13T10:10:16","modified_gmt":"2021-08-13T14:10:16","slug":"blog-considerations-assessing-impact","status":"publish","type":"post","link":"https:\/\/carleton.ca\/tls\/2017\/blog-considerations-assessing-impact\/","title":{"rendered":"Blog: Considerations for assessing impact"},"content":{"rendered":"

By Kevin Cheung, Associate Professor, School of Mathematics and Statistics<\/em><\/p>\n

When instructors make changes in their teaching, they often want to know the impact of such changes.\u00a0 Even though countless studies in higher education exist in the literature, conducting a proper study that meets statistical rigour is hardly a walk in the park.<\/p>\n

As an example, say we want to assess the impact of the choice between two textbooks, Book A and Book B, for a course.\u00a0 How should we set up a study?<\/p>\n

Three immediate options come to mind:<\/p>\n

    \n
  1. Teach the course in a single term in a way that is agnostic to the choice of the textbook and randomly assign students to each of these textbooks as their main reference.<\/li>\n
  2. Teach two sections of the course in the same term using Book A in one section and Book B in the other section with the same set of assessments.<\/li>\n
  3. Teach with Book A in one term and Book B in another term while keeping everything else as similar as possible.<\/li>\n<\/ol>\n

    Option 1 is known as A\/B testing<\/a>. It is attractive in theory but is unimplementable in the current context. For instance, are students willing to be randomly assigned?\u00a0 Can we ensure that students assigned Book A won\u2019t consult Book B or vice versa? (Note that A\/B testing might work in a MOOC<\/a> setting as suggested by a patent application by Khan Academy<\/a> though the conclusions drawn could be weak<\/a>.)<\/p>\n

    Option 2 gets around the difficulty of randomly assigning the two textbooks to a certain extent though it still suffers from the problem that students will know about the two textbooks.\u00a0A potentially bigger problem is that the sections of students might be quite dissimilar. It might happen that one section fits the schedules of students in a particular program while the other does not, possibly leading to confounding<\/a>; any observed difference in outcomes could very well be correlated with the makeup of students in the different sections instead of the textbook choice.<\/p>\n

    Option 3 seems viable if the course is taught in the same term of different academic years with identical course schedules. However, creating assessments consistent across terms is nontrivial. Students taking the course in a later term often can obtain materials from prior terms through friends and \u201cstudy assistance\u201d platforms.\u00a0 Of course, they can also find out about an alternative textbook from these materials.\u00a0In addition, ensuring that the teaching is consistent across terms is challenging.<\/p>\n

    Given the difficulties mentioned above, should we simply give up?\u00a0 Definitely not.\u00a0We just need to be mindful of what factors could be at play and what the potential limitations are. In fact, identifying such factors and limitations can sometimes help us learn more about students\u2019 learning environment.<\/p>\n","protected":false},"excerpt":{"rendered":"

    By Kevin Cheung, Associate Professor, School of Mathematics and Statistics When instructors make changes in their teaching, they often want to know the impact of such changes.\u00a0 Even though countless studies in higher education exist in the literature, conducting a proper study that meets statistical rigour is hardly a walk in the park. As an […]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_relevanssi_hide_post":"","_relevanssi_hide_content":"","_relevanssi_pin_for_all":"","_relevanssi_pin_keywords":"","_relevanssi_unpin_keywords":"","_relevanssi_related_keywords":"","_relevanssi_related_include_ids":"","_relevanssi_related_exclude_ids":"","_relevanssi_related_no_append":"","_relevanssi_related_not_related":"","_relevanssi_related_posts":"","_relevanssi_noindex_reason":"","_mi_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":"","_links_to":"","_links_to_target":""},"categories":[142],"tags":[],"yoast_head":"\nBlog: Considerations for assessing impact - Teaching and Learning Services<\/title>\n<meta name=\"description\" content=\"By Kevin Cheung, Associate Professor, School of Mathematics and Statistics When instructors make changes in their teaching, they often want to know the\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/carleton.ca\/tls\/2017\/blog-considerations-assessing-impact\/\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"CCMS Editor\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/carleton.ca\/tls\/2017\/blog-considerations-assessing-impact\/\",\"url\":\"https:\/\/carleton.ca\/tls\/2017\/blog-considerations-assessing-impact\/\",\"name\":\"Blog: Considerations for assessing impact - Teaching and Learning Services\",\"isPartOf\":{\"@id\":\"https:\/\/carleton.ca\/tls\/#website\"},\"datePublished\":\"2017-06-14T12:51:49+00:00\",\"dateModified\":\"2021-08-13T14:10:16+00:00\",\"author\":{\"@id\":\"https:\/\/carleton.ca\/tls\/#\/schema\/person\/b84c96792cb38e12a65f6fd1220449ad\"},\"description\":\"By Kevin Cheung, Associate Professor, School of Mathematics and Statistics When instructors make changes in their teaching, they often want to know the\",\"breadcrumb\":{\"@id\":\"https:\/\/carleton.ca\/tls\/2017\/blog-considerations-assessing-impact\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/carleton.ca\/tls\/2017\/blog-considerations-assessing-impact\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/carleton.ca\/tls\/2017\/blog-considerations-assessing-impact\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/carleton.ca\/tls\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Blog\",\"item\":\"https:\/\/carleton.ca\/tls\/category\/blog\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Blog: Considerations for assessing impact\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/carleton.ca\/tls\/#website\",\"url\":\"https:\/\/carleton.ca\/tls\/\",\"name\":\"Teaching and Learning Services\",\"description\":\"Carleton University\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/carleton.ca\/tls\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/carleton.ca\/tls\/#\/schema\/person\/b84c96792cb38e12a65f6fd1220449ad\",\"name\":\"CCMS Editor\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/carleton.ca\/tls\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/9690ef8273d7ed6fdf7b62ea06678d07?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/9690ef8273d7ed6fdf7b62ea06678d07?s=96&d=mm&r=g\",\"caption\":\"CCMS Editor\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Blog: Considerations for assessing impact - Teaching and Learning Services","description":"By Kevin Cheung, Associate Professor, School of Mathematics and Statistics When instructors make changes in their teaching, they often want to know the","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/carleton.ca\/tls\/2017\/blog-considerations-assessing-impact\/","twitter_misc":{"Written by":"CCMS Editor","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/carleton.ca\/tls\/2017\/blog-considerations-assessing-impact\/","url":"https:\/\/carleton.ca\/tls\/2017\/blog-considerations-assessing-impact\/","name":"Blog: Considerations for assessing impact - Teaching and Learning Services","isPartOf":{"@id":"https:\/\/carleton.ca\/tls\/#website"},"datePublished":"2017-06-14T12:51:49+00:00","dateModified":"2021-08-13T14:10:16+00:00","author":{"@id":"https:\/\/carleton.ca\/tls\/#\/schema\/person\/b84c96792cb38e12a65f6fd1220449ad"},"description":"By Kevin Cheung, Associate Professor, School of Mathematics and Statistics When instructors make changes in their teaching, they often want to know the","breadcrumb":{"@id":"https:\/\/carleton.ca\/tls\/2017\/blog-considerations-assessing-impact\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/carleton.ca\/tls\/2017\/blog-considerations-assessing-impact\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/carleton.ca\/tls\/2017\/blog-considerations-assessing-impact\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/carleton.ca\/tls\/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https:\/\/carleton.ca\/tls\/category\/blog\/"},{"@type":"ListItem","position":3,"name":"Blog: Considerations for assessing impact"}]},{"@type":"WebSite","@id":"https:\/\/carleton.ca\/tls\/#website","url":"https:\/\/carleton.ca\/tls\/","name":"Teaching and Learning Services","description":"Carleton University","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/carleton.ca\/tls\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/carleton.ca\/tls\/#\/schema\/person\/b84c96792cb38e12a65f6fd1220449ad","name":"CCMS Editor","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/carleton.ca\/tls\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/9690ef8273d7ed6fdf7b62ea06678d07?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/9690ef8273d7ed6fdf7b62ea06678d07?s=96&d=mm&r=g","caption":"CCMS Editor"}}]}},"acf":{"Post Thumbnail":false},"_links":{"self":[{"href":"https:\/\/carleton.ca\/tls\/wp-json\/wp\/v2\/posts\/20005"}],"collection":[{"href":"https:\/\/carleton.ca\/tls\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/carleton.ca\/tls\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/carleton.ca\/tls\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/carleton.ca\/tls\/wp-json\/wp\/v2\/comments?post=20005"}],"version-history":[{"count":1,"href":"https:\/\/carleton.ca\/tls\/wp-json\/wp\/v2\/posts\/20005\/revisions"}],"predecessor-version":[{"id":40144,"href":"https:\/\/carleton.ca\/tls\/wp-json\/wp\/v2\/posts\/20005\/revisions\/40144"}],"wp:attachment":[{"href":"https:\/\/carleton.ca\/tls\/wp-json\/wp\/v2\/media?parent=20005"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/carleton.ca\/tls\/wp-json\/wp\/v2\/categories?post=20005"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/carleton.ca\/tls\/wp-json\/wp\/v2\/tags?post=20005"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}