{"id":237,"date":"2024-04-09T14:22:04","date_gmt":"2024-04-09T18:22:04","guid":{"rendered":"https:\/\/carleton.ca\/xlab\/?p=237"},"modified":"2024-04-16T13:22:48","modified_gmt":"2024-04-16T17:22:48","slug":"the-behaviour-spaces-of-large-language-models","status":"publish","type":"post","link":"https:\/\/carleton.ca\/xlab\/2024\/the-behaviour-spaces-of-large-language-models\/","title":{"rendered":"The Behaviour Space(s) of Large Language Models"},"content":{"rendered":"

By Shawn Graham.<\/p>\n

For many years, I built agent based models of various phenomena based on archaeological data. These are models of interacting software agents in a simulated environment, with many different parameters; the first step in understanding what your model implies for the culture under study is to understand how your model behaves. That is to say, history is run but once but given a set of x,y,z conditions in the past, there were x,y,z possible outcomes. So you run the model hundreds of times at each combination of variables. You account for the randomness, the contingency, of your model by running over and over again, and by doing that for every combination of variables, you end up with a landscape of possible outcomes. (Then, you’d match your actual archaeological evidence against this landscape and see which combination(s) of values map best to your observations, and in this way you’d know something new about the phenomenon).<\/p>\n

So why can’t we do that with large language models? They’re probabilistic! They’re stochastic! Why don’t we run them over and over and over again, attenuating temperature and top-p and other values, so that we end up with a landscape of possibilities for particular prompts?<\/p>\n

I wrote a quick notebook up using the Cohere api. Give it a prompt, and give it a range of temperature and top-p values to test and it’ll generate several iterations of responses. Then, measure the cosine similarity of every pair of responses. Et voil\u00e0, the behaviour space of the command-r model when given the prompt ‘Write a haiku about Bob Dole’. You could write a library of ‘standard’ prompts – without all of the alchemical extras – and see how models compare. What might we find?<\/p>\n

Notebook Link <\/a><- update apr 16 -> there’s something a bit wonky with my code for generating variations, so when I have a moment I’ll delve back into it. But in the meantime, a similar kind of experiment was conducted by Max Woolf on slight modulations to prompts. I’ll probably use his code for generating variations, modify to sweep parameters, then bolt the tf-idf on as a scorer. See his post here<\/a>.<\/p>\n

<\/p>\n

\"\"

Bob Dole Haiku prompt, landscape of results, Command-R model<\/p><\/div>\n

\"\"

Bob Dole Haiku prompt, landscape of results, Command Model<\/p><\/div>\n","protected":false},"excerpt":{"rendered":"

By Shawn Graham. For many years, I built agent based models of various phenomena based on archaeological data. These are models of interacting software agents in a simulated environment, with many different parameters; the first step in understanding what your model implies for the culture under study is to understand how your model behaves. That […]<\/p>\n","protected":false},"author":6,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_relevanssi_hide_post":"","_relevanssi_hide_content":"","_relevanssi_pin_for_all":"","_relevanssi_pin_keywords":"","_relevanssi_unpin_keywords":"","_relevanssi_related_keywords":"","_relevanssi_related_include_ids":"","_relevanssi_related_exclude_ids":"","_relevanssi_related_no_append":"","_relevanssi_related_not_related":"","_relevanssi_related_posts":"","_relevanssi_noindex_reason":"","_mi_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":"","_links_to":"","_links_to_target":""},"categories":[1],"tags":[68],"yoast_head":"\nThe Behaviour Space(s) of Large Language Models - X-Lab<\/title>\n<meta name=\"description\" content=\"By Shawn Graham. For many years, I built agent based models of various phenomena based on archaeological data. These are models of interacting software\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/carleton.ca\/xlab\/2024\/the-behaviour-spaces-of-large-language-models\/\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"shawngraham\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/carleton.ca\/xlab\/2024\/the-behaviour-spaces-of-large-language-models\/\",\"url\":\"https:\/\/carleton.ca\/xlab\/2024\/the-behaviour-spaces-of-large-language-models\/\",\"name\":\"The Behaviour Space(s) of Large Language Models - X-Lab\",\"isPartOf\":{\"@id\":\"https:\/\/carleton.ca\/xlab\/#website\"},\"datePublished\":\"2024-04-09T18:22:04+00:00\",\"dateModified\":\"2024-04-16T17:22:48+00:00\",\"author\":{\"@id\":\"https:\/\/carleton.ca\/xlab\/#\/schema\/person\/e8707158a71e77734ea13346b6e46feb\"},\"description\":\"By Shawn Graham. For many years, I built agent based models of various phenomena based on archaeological data. These are models of interacting software\",\"breadcrumb\":{\"@id\":\"https:\/\/carleton.ca\/xlab\/2024\/the-behaviour-spaces-of-large-language-models\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/carleton.ca\/xlab\/2024\/the-behaviour-spaces-of-large-language-models\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/carleton.ca\/xlab\/2024\/the-behaviour-spaces-of-large-language-models\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/carleton.ca\/xlab\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"News\",\"item\":\"https:\/\/carleton.ca\/xlab\/category\/news\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"The Behaviour Space(s) of Large Language Models\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/carleton.ca\/xlab\/#website\",\"url\":\"https:\/\/carleton.ca\/xlab\/\",\"name\":\"X-Lab\",\"description\":\"Carleton University\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/carleton.ca\/xlab\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/carleton.ca\/xlab\/#\/schema\/person\/e8707158a71e77734ea13346b6e46feb\",\"name\":\"shawngraham\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/carleton.ca\/xlab\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/1b4be5c0f305aa12c7a3dd75ae5c731e?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/1b4be5c0f305aa12c7a3dd75ae5c731e?s=96&d=mm&r=g\",\"caption\":\"shawngraham\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"The Behaviour Space(s) of Large Language Models - X-Lab","description":"By Shawn Graham. For many years, I built agent based models of various phenomena based on archaeological data. These are models of interacting software","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/carleton.ca\/xlab\/2024\/the-behaviour-spaces-of-large-language-models\/","twitter_misc":{"Written by":"shawngraham","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/carleton.ca\/xlab\/2024\/the-behaviour-spaces-of-large-language-models\/","url":"https:\/\/carleton.ca\/xlab\/2024\/the-behaviour-spaces-of-large-language-models\/","name":"The Behaviour Space(s) of Large Language Models - X-Lab","isPartOf":{"@id":"https:\/\/carleton.ca\/xlab\/#website"},"datePublished":"2024-04-09T18:22:04+00:00","dateModified":"2024-04-16T17:22:48+00:00","author":{"@id":"https:\/\/carleton.ca\/xlab\/#\/schema\/person\/e8707158a71e77734ea13346b6e46feb"},"description":"By Shawn Graham. For many years, I built agent based models of various phenomena based on archaeological data. These are models of interacting software","breadcrumb":{"@id":"https:\/\/carleton.ca\/xlab\/2024\/the-behaviour-spaces-of-large-language-models\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/carleton.ca\/xlab\/2024\/the-behaviour-spaces-of-large-language-models\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/carleton.ca\/xlab\/2024\/the-behaviour-spaces-of-large-language-models\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/carleton.ca\/xlab\/"},{"@type":"ListItem","position":2,"name":"News","item":"https:\/\/carleton.ca\/xlab\/category\/news\/"},{"@type":"ListItem","position":3,"name":"The Behaviour Space(s) of Large Language Models"}]},{"@type":"WebSite","@id":"https:\/\/carleton.ca\/xlab\/#website","url":"https:\/\/carleton.ca\/xlab\/","name":"X-Lab","description":"Carleton University","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/carleton.ca\/xlab\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/carleton.ca\/xlab\/#\/schema\/person\/e8707158a71e77734ea13346b6e46feb","name":"shawngraham","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/carleton.ca\/xlab\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/1b4be5c0f305aa12c7a3dd75ae5c731e?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1b4be5c0f305aa12c7a3dd75ae5c731e?s=96&d=mm&r=g","caption":"shawngraham"}}]}},"acf":{"Post Thumbnail Icon":"","Post Thumbnail":false},"_links":{"self":[{"href":"https:\/\/carleton.ca\/xlab\/wp-json\/wp\/v2\/posts\/237"}],"collection":[{"href":"https:\/\/carleton.ca\/xlab\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/carleton.ca\/xlab\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/carleton.ca\/xlab\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/carleton.ca\/xlab\/wp-json\/wp\/v2\/comments?post=237"}],"version-history":[{"count":3,"href":"https:\/\/carleton.ca\/xlab\/wp-json\/wp\/v2\/posts\/237\/revisions"}],"predecessor-version":[{"id":243,"href":"https:\/\/carleton.ca\/xlab\/wp-json\/wp\/v2\/posts\/237\/revisions\/243"}],"wp:attachment":[{"href":"https:\/\/carleton.ca\/xlab\/wp-json\/wp\/v2\/media?parent=237"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/carleton.ca\/xlab\/wp-json\/wp\/v2\/categories?post=237"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/carleton.ca\/xlab\/wp-json\/wp\/v2\/tags?post=237"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}