We live in an age where “we have the most sophisticated information technology in human history yet we have lost the ability to talk to each other”, as Y N Harari famously said. We may blame social media, AI, autocrats, capitalism, or all of those things and more; on the flip side we may look at the past and decide that nothing’s really changed, that maybe a lot of us have always been dumb/racist/selfish, only they wouldn’t show it globally because they didn’t have broadcasting platforms. I don’t know, but I think there’s something each of us can do to increase our own texture, the quality of our thoughts, the knowledge we hold about other places and the people that inhabit them. And as a bonus, we get a deeper appreciation of the value of diversity and a fuller realisation that you don’t have to know stuff, you just have to be open to learn it. Racism and discrimination can and must be fought by the force of the law, but their roots can only be extirpated by openness and recognition.

This list is far from complete, but it’s a start. It is what I try to do myself, but there’s of course plenty more ideas than these.

Listen and read news from outlets from other geographies

The farther from you, the better. Find reputable TV news channels and/or newspapers and magazines from places and cultures far apart from yours. If you’re Western, chances are most of your news intake is from Western sources and while there are of course illustrious outlets that cover the whole world, the base perspective can nevertheless be… Western. Mixing up and listening to the same piece of news from other places can be immensely educational. Many outlets from all over the world have English versions.

Read authors from other countries and less privileged backgrounds

Read books in general, especially the great literature of the world. Investigate great authors from countries you know little about and read their production. If you’re able to, read in the original language (but otherwise translators usually do an amazing job!). I found this person who’s reading a book from each country in the world and I think it’s a delightful project. Also, read women authors, black authors, queer authors, politically-persecuted authors, authors from backgrounds and walks of life that are far from yours. Read quality thoughts about race, freedom, systemic discrimination, get deep into what it can mean to be treated like you’re a lesser human being.

Use your public library (if you have the privilege to have one, I realise not everyone does), frequent your local bookstore: they’ll likely also hold events and presentations. If you’re a bookworm, entering a library can be like being in the quintessential candy store - there’ll be shelves with new acquisitions, shelves organised by topic, shelves celebrating a particular author or theme. Explore.

Watch movies and shows from [same as above]

The exact same as above applies - make an effort to look for the good stuff from abroad. Maybe you have a nearby cinema that regularly hosts foreign film festivals. Or if you pay for a streaming platform there may be sections of material from various countries, it may be worth having a look next time you’re in doubt as to what to watch.

Learn a new language

In the age of machine translation, is learning a language still a worthy enterprise? I’m convinced it is, not much or not necessarily because of practical repercussions, although being able to ask things and sustain a conversation without the need for a phone mediator is clearly empowering, but because it helps widening your mind, and by a good extent. Language is culture and it is history. Of course becoming fluent would help a lot with the two above ideas, but that’s not what I mean here - you don’t have to be fluent to enjoy how different languages may structure things differently or to see that grammatical concepts do not necessarily overlap 1:1 (and in fact, the farther apart the languages, the smaller the chances that they do). For example, Italian has a conditional mood for verbs, which German renders with Konjunktiv II; Mandarin Chinese does not “conjugate” verbs, or even have the same strict concept of a “verb” like in European languages.

You also don’t need to be fluent to make connections between words, explore possible common roots and intertwine the development of a language with the history of peoples who migrated, mixed, were conquered. Some languages don’t share the same alphabet or writing system, some do but with variations. Linguistics is fascinating and it follows the very human history itself - getting some familiarity with it is a wonderful way to embrace the complexity of humanity.

Study some maths

While on the topic of learning, learn mathematics. Maths powers a lot of things - we’d have no tech, no AI, no coding, no bridges and no bookkeeping without it. But we’d also struggle to understand much of the wonder of the world, like for instance how penguins find the most efficient configurations when battling a storm.

Maths is the ultimate logic-trainer. There’s just nothing else like it. Go to the basics, train your brain, it’ll help in so many ways. You don’t need to enroll at the university or buy expensive textbooks, you just need to have the curiosity and start small - there’s plenty of great, free and entertaining material online. Mathematics is powerful and it is beautiful and as many said, it is more often than not butchered when it is taught. It is a giant shame. However learning it does require focus and some effort. Is it hard? Yes, but the reward is huge. As they say, no pain, no gain - and this applies to pretty much anything in life.

Ask people, be painfully curious

When meeting someone that comes from the other side of the world, it is very easy to keep the conversation on the superficial level of the “oh great/oh I want to go there/oh I was there last year”. Go deeper, be curious (without being creepy of course!): ask them to recommend something, their favourite dish or one they don’t like, what do families and friends do when they spend social time together… get that texture, engage, learn.

It is true that the world is on fire right now, that consumerism has fully shown its ugly face, that places that prided themselves as the proverbial beacon of democracy (?) are turning autocratic, that we are all more digitally connected yet more actually lonely. But it doesn’t have to be this way, and it’s up to all of us to do something to keep educating ourselves, show kindness, connect with those different from us, be humble and learn. Maybe if more and more of us did this we’d naturally attract more people to do the same and we’d actually make a dent.

• A Scanner Darkly, P K Dick
• The Testaments, M Atwood
• The Man in the High Castle, P K Dick
• Brave New World, A Huxley
• Do Androids dream of Electric Sheep?, P K Dick

They’re all beyond excellent - This blog pens some thoughts about the first one. Warning: There are spoilers here, so if you want to read the book you know what to do.

I’d never read anything by Dick before and I can’t remember how I ended up starting with this one (it was probably a recommendation from somewhere) but it was brilliant: themes are the surveillance state, substance addiction, personal identity and the subtle tension between truth and fabricated piles of lies. The book is intensely symbolic and multilayered.

The title

The title is inspired by a biblical passage, possibly taken from the King James version of the Bible (a translation commissioned by King James VI/I of Scotland/England & Ireland):

“For now we see through a glass, darkly; but then face to face: now I know in part; but then shall I know even as also I am known.”</br> – 1 Corinthians 13:12 (King James version)

The passage reflects on the fact that while at the moment we don’t see and understand clearly, we will at some point (in the afterlife?). Here’s the identity theme front and center.

The setting

This wasn’t clear for me immediately but it gradually emerged with reading: we are in a violent authoritarian society, in an unspecified near future, near enough that it would preserve common traits with Dick’s present (the book was written in the ’70s) but far enough that it would exacerbate them into a sinister atmosphere. The main factor is a widespread drug addiction, which the government fights via several branches of law enforcement and (purported) addict rehabilitation programs.

It is also a heavily consumeristic society flattened onto the repetition of the same offerings over and over again, a setting that must have been very much inspired from reality. Two of my favourite quotes:

In Southern California it didn’t make any difference anyhow where you went: there was always the same McDonaldburger place over and over; like a circular strip that turned past you as you pretended to go somewhere.

Life in Anaheim, California, was a commercial for itself, endlessly replayed.

There’s a pervasive sense that nobody knows who others really are: addicts (the “dopes”) lose perception of themselves and others, but non-addicts (the “straights”) are also partially in the dark with respect to identities. A significant element is the special suit that governmental narcotic agents (the “narcs”) wear in order to conceal their identity to everyone (including other narcs) - the suit hides someone’s features by showing a rotation and overlap of random faces: narcs have no identity, they are slaves to the system as much as dopes are slaves to the drugs.

The most popular drug is “Substance D” (never made explicit but likely standing for “death”, though it could also be in combination with “destruction” or “damage”), which has extreme addiction power and ability to destroy: users will gradually lose use of their brains because the connections between hemispheres get severed. People get onto weird obsessions (the book opens with a guy who’s convinced there’s insects all over him and his place and takes a shower after the other). The all vibe is, appropriately to the title, dark.

The protagonist Bob is a narc, but he’s also a user, because he’s undercover and needs to fit in. He thought he was above abuse, but he’s not - he gradually turns into one of the same people he’s supposed to investigate. Bob is tasked with discovering who produces Substance D, as it is known there’s only one root source and the drug is organic.

There’s lots of clever turns of the story, like when Bob, who lives in a house with other dopes, finds himself in the condition to have to investigate himself - the narcs have installed cameras all over the house, they know there is a narc in there but not who’s who, and Bob has to watch the recordings and report, reporting himself. His degradation follows a spiral, and he does realises he’s losing his cognition, medics give him tests which prove it, yet he has to keep going. Who is who becomes even more mixed-up.

Bob’s surname, Arctor, reminisces of “actor”, perhaps a reference to his acting of himself.

Dick was a heavy drug user himself. In the appendix he writes

This has been a novel about some people who were punished entirely too much for what they did.

There is no moral in this novel, it is not bourgeois […] it just tells what the consequences were.

If there was any “sin”, it was that these people wanted to keep on having a good time forever, and were punished for that, but […] the punishment was far too great

There’s also a list of names with the “punishment” they received, in which a “Phil” with “permanent pancreatic damage” is himself.

The language

The rhythm is fast and the prose is raw, at traits jargony, the conversations reported are often trippy, which is a testament to Dick’s writing ability: despite the addicts’ nonsense (words and actions), you as a reader never fail to understand what is happening.

There’s lots of science-like talk about the human brain too, which proves research.

Then, there’s some German quotes, which is a constant in Dick. He appreciated philosophy, including German thinkers, and understood the language.

It’d be largely reductive to brand him as “just” a science-fiction writer. There are plenty of cultural allusions and references you can pick up, the more I read reviews and opinion pieces about this book the more I see them myself.

The ending

Bob ends up in a rehab clinic, one of those cruel government-sponsored places for the supposed recuperation of addicts. Except that they get enslaved instead. He’s been “betrayed” by his girlfriend who is in fact a upper-level governmental agent herself. In the very final scene, while there in a working farm, in a rare epiphany now that his brain is completely gone to mush, he finds the source of Substance D (some flowers) and hence realises that the government itself is the producer.

Authoritarian regimes always need control systems, and control systems need to control each other. It is a chain of power where each element only has partial information and acts on a limited sphere. The narcs are actually a cover-up the government uses to pretend to be on the side of citizens and have superior moral status while in fact they, unbeknownst to themselves, are just pawns, serving the purpose of pretend law-enforcement and really doing the dirty job of citizen surveillance. This has Soviet-style elements and points of connection to Orwell’s 1984.

The oligarchs in power perpetuate a rich-gets-richer way of living and for that it is essential to both create the addiction problem and make sure it can never be solved, while at the same time pretending to tackle it. Everyone else is unknowingly playing into the game. This is redolent of Pain Killer: An Empire of Deceit and the Origin of America’s Opioid Epidemic.

The derived movie

There is a movie (2006) but IMHO its only worth of note element is the fact that it’s fully rotoscoped. Interesting visuals, but it’s not up to the book at all. It felt to me like it was made into a “story” stripped of all the deeper elements and reduced to a quite shallow rendition. But not everyone agrees.

I found 102 of them, currently active - there’s probably more depending on how you count, with merges and renamings, and I’ve collected data about their year of creation, state and founder.

At the age of 16, Wright moved from her native town of Talbotton, Georgia to Tuskegee, Alabama after learning of the opening there of a new university for black students, Tuskegee University, founded in 1881 with the name of Tuskegee Normal School for Colored Teachers. The year was 1888 and she enrolled there.

Wright was the child of formerly enslaved parents, an African-American man and a Cherokee woman. The family was poor - conditions were hard for black folks, especially in the South. She likely used trains for the ride to Alabama as the railway was the most popular mode of transportation for long distances at the time: a train journey in 1888 costed circa 2-3cents/mile, which makes for a total expenditure for Wright between $2 and $3, using an estimate of a total of 100 miles covered. In today’s money, that’s a ballpark of $60-$100 - likely a lot of money for someone of Wright’s background. It is very interesting to note that the same train journey today wouldn’t be possible - several railway lines that used to operate in the area have been discontinued or operate just as freight services, like for instance the Western Railway of Alabama.

She worked as a servant/domestic during the day, frequented courses at night and found herself mentors who would support her education, eventually graduating in 1894. Then she moved to South Carolina, held teaching jobs and decided to give back to the community by founding schools for black students herself, with the special aim to provide industrial and agricultural education: she was keen to equip black folks with the abilities to succeed in life beyond the menial jobs they had been enslaved to do before abolition. Things really didn’t go well initially: not only did she encounter huge resistance by many white people, but several of her attempts at building schools were burned down. Until the one in Denmark, with an early name of Denmark Industrial School, succeeded. It was 1897 and the school started as a single room on top of a grocery store. She then secured a generous donation from benefactor Ralph Voorhees, a wealthy local man with an interest in spending his large fortune in good causes, which enabled the school to move to a new location and expand, and eventually take his name (it was Wright’s decision to do so). Since then, the school has offered education to many people, remains predominantly black-frequented and has largely expanded its offering in terms of programs and degrees.

Raised in the Episcopal faith, religion has been a strong driver behind Wright’s ambition; the church, as well as individual figures, have also supported her all along and financed her endeavours. She travelled a lot around all her life, often on foot, to raise money and also awareness and her legacy remains huge.

Of a sick disposition all her life, she died aged only 34 in 1906. Her grave lies on the university campus.

Sources

I’ve used a combination of generative AI tools to aid the collecting of information about this, specifically chatGPT and Gemini with Deep research. I’ve checked information and sources for reliability.

• Wikipedia page for E E Wright
• Page about her on the Voorhees University website
• Voorhees University documentary about E E Wright
• A brief timeline of HBCUs on UNCF
• E E Wright on the South Carolina Encyclopedia
• The Nomination Form of Voorhees University on the National Register of Historic Places
• There’s a marker plaque in Talbotton about E E Wright

The amount of research appearing is on a clear upward trend and some fields are particularly popular, especially, and maybe unsurprisingly, Artificial Intelligence. It’s really hard, if not virtually impossible, to keep up with the pace, so we built something that could help us distill and easily find information.

What is it, what can it do

Askademic is a CLI tool - we’ve chosen to start with the path of least resistance, we may create a UI later on, maybe when the tool will have more features. It’s powered by PydanticAI as its agents framework and currently based on Gemini (2.0 Flash) as its underlying LLM, though we’re shipping support for other families too.

Askademic can:

• summarise recent papers (those from the latest available day) published in a arXiv category
• find specific information in the literature by answering questions
• talk about a given paper

All instructions are on GitHub but more details on what it can do and how are on the excellent DeepWiki (which IMHO is itself one of the best applications of LLMs).

The whole process of developing agents and orchestrating them is made very easy by PydanticAI, which has some rough edges being a juvenile project itself, but it is overall really good.

Science or pun

Your question first passes through an agent (“allower”) that decides whether you’re asking something scientific or not - in the latter case, it gives you a pun related to what you asked, e.g.:

Other things I’ve seen are

What is the capital of Italy?
I "Rome" around looking for science in this question, but it's just geography!

What is the meaning of life?
I'm not sure, but I bet it's not something you can find in a test tube!

… and so on.

Orchestrating

Passing this first layer there’s an “orchestrator” agent that triages the request to either of 3 sub-agents, depending on its nature: if you ask for summaries of recent papers by category it’ll go to a “summary” agent, a particular research questions will be given to a “question” agent, anything about a specific paper, given by title or URL go to an “article” agent.

The orchestrator has memory (message history), so to make use of what was discussed before in case you want to ask for follow-ups (which is particularly useful for the second use case, the questions one). The memory feature, courtesy of PydanticAI, works of course by attaching all previous messages exchanged to the prompt so that the LLM has context. This is where Gemini comes in handy with its 2M window context - it’s pretty hard to exceed it. We are adding support for Claude and one of the pain points is the limited (in comparison) context window - we are still experimenting with the best choices.

Summarising

Summarising works by using the taxonomy of arXiv categories and is based on abstracts. We feed the categories as hardcoded, each one is given as ID-description pair (e.g. “hep-ex” - “High Energy Physics - Experiment”). We were originally scraping the arXiv taxonomy page but that obviously proved to be the best way to get banned by arXiv (that is, to get given a captcha wall). Because there isn’t endpoint for categories, we resorted to just hard-coding the list.

The summary agent is instructed to:

1. first, choose the most relevant category according to your request
2. then, run a first API query to figure out what’s the most recent available day of publications
3. finally, do another query for all abstracts of papers published in that category, which we then curtail to the ones belonging to the day from step 2.

We say all abstracts because we use a max of 300: this comes from a heuristic, the max we’ve seen published in a category in a day is below that mark so it seemed sensible, and a cap is useful not to explode the context too much anyway. Certainly a point of improvement. We use abstracts rather than full papers for the same reason, they’s add too much text.

You can e.g. ask

Give me the latest papers in group theory

and you get this kind of output:

category: category_id='math.GR' category_name='Group Theory'
latest_published_day: 2025-05-08
summary: The latest research covers several topics in group theory and related areas.
One paper focuses on proving a fixed point theorem for the action of $SL_n$ over local
fields on symmetric spaces of infinite dimension and finite rank
(http://arxiv.org/pdf/2505.05220v1). Another paper deals with the conjugate generation
of sporadic almost simple groups, determining a specific value related to their
automorphisms (http://arxiv.org/pdf/2505.05173v1). There is also a classification of
proper partial linear spaces affording imprimitive rank 3 automorphism groups,
including the construction of infinite families and individual examples
(http://arxiv.org/pdf/2505.05124v1). Further research characterizes nilpotent bicyclic
groups, generalizing a previous result related to abelian bicyclic groups
(http://arxiv.org/pdf/2505.05065v1). Finally, one article studies flag-transitive
automorphism groups of 2-designs with prime λ, identifying specific designs with
exceptional or sporadic simple groups as socles (http://arxiv.org/pdf/2505.04985v1).
recent_papers_url: https://arxiv.org/list/math.GR/new

Answering questions

This agent answers questions by looking in the literature. It’s made in such a way to:

1. decide what queries to run agains the API, searching for matches within the abstract
2. run said queries and collect abstracts
3. evaluate all abstracts for relevance to the question and only pick the most relevant ones
4. finally, read the papers related to those abstracts (but we curtail the number of them)
5. produce an answer with all this info

For 1, it works pretty well because the arXiv API is based on Apache Lucene, the relevance retrieval is pretty good.

The paper retrieval is done by pulling from the site because the whole paper text isn’t reachable via API - this is something we need to improve to as again you may get locked out.

An example, with question How is the performance of LLMs measured when it comes to mathematical reasoning?

response: LLMs\' performance in mathematical reasoning is measured using various metrics and
benchmarks. Here\'s a breakdown of how the articles describe it:

*   **Metrics**: The primary metrics used are Accuracy (ACC), Reasoning Score (RS), and Clarity Score
(CS), reflecting different dimensions of mathematical understanding and communication
(http://arxiv.org/pdf/2503.10573v2.pdf).
*   **Benchmarks**: Several benchmark datasets are used to evaluate LLMs, including Math Competition
(MATH), Grade School Math (GSM8K), and Massive Multitask Language Understanding (MMLU) math subset
(http://arxiv.org/pdf/2503.10573v2.pdf).
*   **Evaluation Framework**: A process-oriented framework is used to evaluate LLMs\' ability to
construct mathematical models, using solvers to compare outputs with ground truth
(http://arxiv.org/pdf/2405.13144v3.pdf).
*   **MATHHAY Benchmark**: MATHHAY is an automated benchmark designed to assess the long-context
mathematical reasoning capabilities of LLMs. It includes questions of varying difficulty levels to
assess LLMs’ reasoning abilities across different input lengths (32K, 64K, 128K)
(http://arxiv.org/pdf/2410.04698v1.pdf).

article_list: ['http://arxiv.org/pdf/2503.10573v2.pdf', 'http://arxiv.org/pdf/2405.13144v3.pdf',
'http://arxiv.org/pdf/2410.04698v1.pdf']

Note that these are the API queries it ran (sorting parameters are not left for the LLM to choose, they’re imposed):

http://export.arxiv.org/api/query?search_query=abs:llm mathematical reasoning performance metrics&start=0&max_results=20&sortBy=relevance&sortOrder=descending
http://export.arxiv.org/api/query?search_query=abs:evaluating mathematical abilities of large language models&start=0&max_results=20&sortBy=relevance&sortOrder=descending
http://export.arxiv.org/api/query?search_query=abs:mathematical reasoning benchmarks for llms&start=0&max_results=20&sortBy=relevance&sortOrder=descending
http://export.arxiv.org/api/query?search_query=abs:llm performance on math problems&start=0&max_results=20&sortBy=relevance&sortOrder=descending
http://export.arxiv.org/api/query?search_query=abs:measuring mathematical understanding in llms&start=0&max_results=20&sortBy=relevance&sortOrder=descending

Talking about an article

This agent really is quite simple. It retrieves a paper by link or title, “reads it” as whole text (although we also curtail the input here, to a max of chars) and responds to whatever you asked.

For instance, asking what was the point of paper 'attention is all you need'?

will lead to something like

response: The point of the paper 'Attention is all you need' is to introduce a new and simple network architecture called the Transformer. This architecture relies solely on attention mechanisms,
dispensing with recurrence and convolutions entirely, to draw global dependencies between input and
output. The paper demonstrates that the Transformer is superior in quality, more parallelizable, and
requires less training time compared to complex recurrent or convolutional neural networks, achieving
state-of-the-art results on machine translation tasks.
article_title: Attention is all you need
article_link: http://arxiv.org/pdf/1706.03762v7

Reducing agency

Initially, each of the agent was doing all the steps on its own in a full pipeline. They had been conceived as single agents, with a detailed prompt giving all instructions at once.

We’ve now moved to a system where the summary and question agents are actually cascades of small agents each doing a step. Some of the steps may not be “agentic” at all, for example for the summary agent, there’s first an agent that chooses the matching category, then some other code runs the API queries (as opposed to an agent equipped with a tool doing thi), finally passing the results an agent that summarises the results. Similarly applies to the question agents, small agents in cascade “decide” what API queries to run, judge relevance of abstracts and then summarise the whole thing from the papers retrieved.

Why this? Because we realised that sometimes the pipeline as executed the whole way by a single macro-agent would go on forever, it would still run queries even when the answer/response was clearly there. We had to remove some agency!

This process is ongoing and we’re experimenting with the best compromises, also noting that there may be differences depending on what the LLM family is. For example, we’re currently in the process of reducing the agency of the “article” agent too, because when asked for a paper that doesn’t exist it tends to go on forever trying to find it. The old adage of do things the easiest and most reliable way, do not complicate your life adding uncertainty, still stands strong. Agents/LLMs are great for things that you can’t really do otherwise, things like searching for info within text or creating summaries or matching up content, but if they have too much to do at once the chances that they bomb it may not be worth the risk.

It is known that LLMs tend to be yes-minions, they’ve been trained to produce answers for a positive query (as in, when the answer exists) rather than for judging whether the query makes sense in the first place. We are learning about when it is best to leave them to decide things on their own and when a pipeline of intertwined agents/IFTTT statements suits our needs much better instead. This point will likely be explored in another post, hope you enjoyed this!

So I thought that it may be a great idea to go a bit meta (not the company, the original meaning) and use GenAI to summarise papers which will mostly be …about GenAI. Summarising text is one of the most popular use cases for LLMs, and a growing area of research deals with their effectiveness on this, see this relatively recent review paper.

Wanting to also try out the Gemini LLM family, I’ve decided to build a lil’ tool - which for now is nothing more than a Colab notebook (see notes below) - that parses the AI “recent papers” page from the ArXiv and spits out a digestible summary of all the papers published on the latest available day.

The ArXiv was built in the early 1990s by the physics community as a way to share preprints and gather feedback before submitting them for publication to a journal. It has since grown to host a plethora of disciplines and sub-disciplines and is the main general reference for scientific “papers” - the quotes are because what’s there are preprints, largely material that’s not been peer-reviewed, so always to take with a pinch of salt. The ArXiv is however undoubtedly a great manifestation of the power of open science/access/source, and allows lots of people, not just scientists but software engineers, practitioners and just about anyone to access knowledge as it comes. It doesn’t have everything but it has a lot.

For AI in particular, I have the feeling that it is the main reference. Thankfully each discipline is organised in classes and subclasss which each have a “recent entries” page with listings per day, from Monday to Friday. I don’t know for a fact but wouldn’t be surprised to learn that the AI subclass (subclass of Computer Science) is the most popular these days - the last few days have exceeded the 150 papers/day. Physics as a whole class has seen 49 entries on Feb 14.

Procedure

My Colab notebook is here, versioned on Github here. My idea for now is to run this daily, check out what it comes up with and maybe later on move it to an automated tool (it can be something as simple as a cron job or scheduled on the cloud, maybe allowing it to send myself daily emails with the summaries).

The whole idea is to let Gemini “read” papers published to the ArXiv in the latest available day and summarise them in a succint, clear and to-the-point way. Of course, there is no substitute for reading a paper yourself, understanding and appreciating it in detail. Further, LLMs can hallucinate, fail to point out the main points or just do a bad job, I’m aware. But still, caveats considered this is very valuable to me because I wouldn’t be able to ingest all the daily info myself - it’s been a pain point for a while.

I’ve used this prompt, which seemed to work well after a few anecdotal tests:

sys_instruct = """
                You are an experienced reader of academic literature and
                an expert in distilling important findings in a way that is understandable and clear.
               """

prompt = """This is a paper on AI. 
            Summarise its results in 3 lines, avoiding obscure jargon and going to the point.
            If there are valuable examples that aid understanding, report them in a nutshell.
            """

The paper will be given as a file directly, without the need to first extract its text - this is easily doable via the Gemini API.

You may have thought “why don’t you just read the abstracts that the ArXiv aggregation page already presents”? Well, often they’re not enough, they don’t necessarily present a clear idea devoid of jargon that you’d only learn when reading the whole thing. Also, I am very keen on getting the example(s) in the summary, they significantly speed up my understanding. But I can test prompts to check how much quick info that’s not just the abstract I can get - the more I run this the more I’ll see.

I initially thought of using an LLM to parse the “recent papers” page but decided against it as it felt like a waste: it is easy enough to extract info with a bit of good old requests and BeautifulSoup. The page is well structured and I don’t expect it to change in time, so it’s a case of finding the ordered list of paper IDs, URLs and titles. After figuring out what’s the latest day of data available and how many entries it has (from parsing the first H3 header, e.g. “Fri, 14 Feb 2025 (showing first 50 of 159 entries )”), this is it:

# find the URLs to these papers for the latest day only (up to n_entries as per above)
paper_links = soup.find_all("a", {"title": "Download PDF"})[:n_entries]

# Extract the paper IDs and links
paper_ids, paper_urls = [], []
for link in paper_links:
    paper_url = "https://arxiv.org" + link["href"]
    paper_id = link["href"].split("/")[-1].split("v")[0]  # Extract the ID

    paper_ids.append(paper_id)
    paper_urls.append(paper_url)

# separately, find all titles (this is due to how the DOM is structured)
# they'll appear in the same order so order counts
paper_title_divs = soup.find_all("div", {"class": "list-title mathjax"})[:n_entries]

paper_titles = []
for title_div in paper_title_divs:
    paper_titles.append(title_div.contents[1].split('\n')[1].lstrip())

Then, I download all the papers locally (to the file system Colab is giving me, which, note, is ephemeral) with urllib.request - this is very fast, takes about 20 seconds for 150 papers (files are rather small).

Finally, I can ask Gemini to process them with the prompt above (and ingesting the file itself), one by one with:

client = genai.Client(api_key=userdata.get('GEMINI_API_KEY'))

# configure which Gemini to run 
model = "gemini-2.0-flash"

file_ = client.files.upload(file=f'pdfs/{filename}')
start_time = datetime.now()
response = client.models.generate_content(
    model=model,
    config=types.GenerateContentConfig(system_instruction=sys_instruct,
                                       temperature=0),  # use greedy decoding
    contents=[prompt, file_])

I’ve used Gemini 2.0 Flash but I’ll check out if Lite can do the same job and maybe faster. About the API key, you can store it in the “secrets” feature in Colab which works a bit like environment variables. Gemini has a free tier of 15 requests/minute which I’ve not exceeded with my daily run, because a document gets processed in about 5 seconds (median across all), which means in a minute I can expect to send ~12 requests. I’ll see how it plays out in time, more than happy to pay a little for this, pricing is quite friendly too. I may be wrong but Gemini is the only proprietary (non open-source) LLM family with a free API tier.

I save the summaries as they come, keep track of tokens consumed in input and output and latency just for the sake of having stats. At the end, I generate a lil’ HTML page with the summary, title and href to URL of each paper (I’ll spare you the code for that here, check the notebook in case). For Feb 14, the summary page looks like this

It’s bare, but I don’t need anything else - all I need is reading/skimming it every day and, if desired, go check out the paper itself (hence the links are important). I could make it send it to me via email I suppose.

What can be improved

A lot of things. What comes to mind immediately:

• The design of the HTML summary page!
• Send the summary somewhere, e.g. via email
• Add a preliminary check to see if the latest available day of papers has been already summarised
• Automate it as a job - for now a daily manual run is good enough while I check results out
• A better prompt?
• The Colab runtime has a time cutoff - for now running the notebook to summarise ~150 papers takes about 20 mins overall so it’s well within limits (of course the slowest part is the LLM call, but Gemini seems fast enough)
• The LLM doesn’t have to be Gemini (although I have to say I quite like it!), I could experiment with other ones, and I’d particularly love to use open source ones

New additional “features” I could add

AI papers don’t get published just under the “AI” subclass, another popular one is e.g. “Computation and Language” - I could do the same for each section of interest.

Inevitably, these’d be too many papers. I could then add a second layer than collates all the daily paper summaries and gives me an overview of the themes.

Let me know what you think - find me on BlueSky or LinkedIn (see sidebar). Any feedback is greatly appreciated, including if you find mistakes.

Also, if you’re one for newsletters, my post can get regularly sent to you:

It is remarkable that a science which began with the consideration of games of chance should have become the most important object of human knowledge.
Pierre-Simon, Marquis de Laplace, Théorie Analytique des Probabilités, 1812

The developments of gambling and probability theory have gone hand in hand throughout history: humans have been playing dice and cards for centuries, humans have then started formalising strategies for wins and losses.

Primitive forms of dice are among the first playthings invented by humanity; card games were likely invented in ancient China in the 9th century. There are references to games in numerous pieces of literature, for instance the “Zara” dice game, from the Middle Ages, is mentioned in Dante Alighieri’s Comedy and other works. The Zara game is very simple: you throw three 6-sided dice at once after choosing a number 3-18 (respectively the minimum and maximum achievable) - because results in the middle of the scale are obtainable with a larger number of dice combinations (you get a 3 with 1-1-1 only, a 18 with 6-6-6 only, a 4 with 1-1-2/1-2-1/2-1-1, a 17 with 6-6-5, 6-5-6, 5-6-6, …, but a 10 or 11 with 27 combos each, out of the 216 possible ones), obviously it’s convenient to bet on them. It does feel like an uninteresting game because of this, but maybe they hadn’t figured it out yet back then! This game is very useful as toy example to teach concepts in probability.

Starting from the 17th century a whole new area of mathematics developed to investigate and formalise the concepts and measurements around chance, odds, wins and losses. Fermat, Pascal, Huygens, Bernoulli, Cardano, de Moivre, Laplace are some of the famous names involved in the founding of this new intellectual endeavour. I’d love to know if the pioneers of probability were also gamblers themselves, but I’m not sure I found reliable information. Either way, it was during their times that playing and gambling took on a whole new level in Europe. It is not uncommon anyway to have people involved in mathematics and forms of gambling or investing to this day - Jim Simons is a brilliant example: a mathematician by background and career with important academic contributions and prizes won, he also worked at the NSA and eventually launched a hedge fund company which made him a billionaire. He’s also been a great philantropist who contributed all his life, monetarily and with time and energy to causes related to the dissemination of scientific education and research, establishing and funding, amongst other nonprofits, “Math for America”.

We’ll give a brief overview of something from the early era of probability theory: Bernoulli trials and the related probability distributions.

The Bernoulli distribution

Let’s consider a binary variable \(X\), that is, one that can take only 2 values, say “1” (usually representing success) with probability \(p\) and “0” (usually representing failure), with probability \(1-p\). Phenomena behaving like this are known as Bernoulli trials - examples are a coin flip or randomly asking people in the street whether they had cereal for breakfast or not.

We can write the probability mass function (the mathematical form of probability for each value) as

because when \(x=1\) we are left with \(p\) and when \(x=0\) with \(1-p\). This is the Bernoulli distribution.

Expected value (the mean where each value is weighted by its probability) and variance (measuring the spread of values) calculate respectively as

and

The Bernoulli responsible for this formalisation was Jakob (there are many famous Bernoullis), son of a pharmacist and brother to Johann. The boys were a bit naughty and disobeyed their father who wanted them to study medicine and theology and did mathemathics instead. We must be very thankful they did as they left us with a huge corpus of important work. Also, Johann’s sons and their sons continued the mathematical dynasty: the Bernoulli family was quite gifted for sure.

More successes and more failures: the Binomial

In this business it’s easy to confuse luck with brains.
Jim Simons [undocumented]

The Bernoulli is a special case of a more general distribution, the Binomial: you have \(k\) successes in a set of \(n\) Bernoulli trials. The probability mass function writes as

The first element, \({n}\choose{k}\), quite uncreatively called the “binomial coefficient”, tells you how many ways of creating groups of \(k\) from \(n\) values are there and writes as (with all expansion)

If you have a bag with 3 balls, named say A, B and C, and want to extract 2 of them (at the same time, that is, with no replacement), you can pick any of 3 different sets: AB, AC, BC, \({3 \choose 2} = \frac{3\cdot2}{2}=3\). If the balls are 4 (A,B,C,D) and you still want a group of 2, you can pick AB, AC, AD, BC, BD, CD, which makes 6 groups or \({4 \choose2} = \frac{4\cdot3}{2\cdot1}\). For group of 3 with 4 balls you can do ABC, ABD, ACD, BCD, \({4\choose 3} = \frac{4\cdot3\cdot2}{3\cdot2\cdot1}\). And so on.

Expected value and variance of the Binomial calculate respectively as \(\mathbb{E}[X] = n p\) and \(Var[X] = n p (1- p)\) (see Wikipedia for the proofs).

Let’s see this with a bit of code.

First the imports and setting up of a pseudo-random number generator in Numpy:

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng()

then we simulate 1000 trials of a fair coin flip, that is, one where each face is equiprobable (probability 0.5)

n = 1000    # choose how many trials
trials.append(rng.choice([0,1], size=n))   # numpy.choice will by default use a uniform distr over values

and we count successes and their probability

successes = np.cumsum(trials)
p = successes / np.arange(1, n+1)

which we can plot

plt.plot(np.arange(1, n+1), p, label='p_success', c='g')
plt.plot(np.arange(1, n+1), 1 - p, label='p_failure', c='r')

plt.grid()
plt.legend()

plt.title('Probability of success and failure')
plt.xlabel('Number of trials')
plt.ylabel('Probability')

obtaining these trends:

If we vary the probability of success (using arg p in numpy.random.choice), so to represent unfair coins, we get these trends

The Binomial distribution generalises into a Multinomial when possible results are more than two.

Probability theory is full of nice and fun questions.

Good reads

• Probability and Statistics on Encyclopedia Britannica
• J MacDonald, The ancient origins of Dice, JSTOR Daily 2018 - with lots of links to archeology studies
• J Roebke, Putting his money where his math is, Seed Magazine (defunct) 2006, via Web Archive - nice article about J Simons and his passion for maths education
• D T Max, Jim Simons, the numbers King, The New Yorker 2017
• J Eglin, How the 18th-century ‘probability revolution’ fueled the casino gambling craze, The Conversation 2024
• J I García-García, N A Fernández Coronado,E H Arredondo, I A Imilpán Rivera, The Binomial Distribution: Historical Origin and Evolution of Its Problem Situations, Mathematics 2022 (open access)

German builds some of its verbs via prefixes, which either add nuance to the original meaning or alter it completely. Those in the latter class are the hard ones to absorb, at least for me. Out of struggling to distinguish my aufmachen from my ausmachen, I started this exercise: at the very start, I made a list of all verbs I could think of whose meaning and usage I was sure of; now, I just add one every time I feel like I’ve learned it, as in, I’ve stored it in my (hopefully) long-term memory and I can comfortably use it.

The whole of German is wonderfully Lego-brickable, but I only do this for verbs because with no other part of speech have I had such struggles, and also because without a rock-solid foundation in verbs it’s hard you’ll be able to build coherent discourse.

I am between B1-B2 levels now and the below is what I’ve got. It doesn’t look like many, but there will be several (many?) verbs that I somewhat know but I’m not really sure about. Also, there may will be verbs I forgot to add. And, I am not tracking all possible easy combinations from a root (the first class from above): I haven’t added wegrennen (run away) for instance, despite understanding it very well and intuitively.

These days the adding-to-list frequency is quite low, maybe I’m approaching the stage where I need to level up and/or diversify sources to have a new burst of novelty like in the early glorious days of this.

S really is queen! We have all the sein, singen, stattfinden, schencken, schicken, … There’s 0, nichts, for C - If I really think hard now, nothing more than an English loan like chatten, should it exists at all (not sure), comes to mind. X and Y have the same fate but that’s not surprising, most words there will be actual loans. Only 1 entry for J, that’s joggen, also a loan.

Likely, and this is important, my verbs are the most common ones (because I’m a learner and don’t yet consistently consume texts aimed at natives) so this isn’t necessarily representative of the whole vocabulary of German verbs.

Can I pull this data from actual German?

What if I want to compare my totally personal German verbs distribution with one derived from German general texts, not mostly those aimed at learners? There’s lots of sources I could use for this but I thought I’d query the mighty Wikipedia. Oh, how lucky, Python has a great package that lets you do just that easily-peasily. And spaCy has models trained for German (and many other languages), that I can use for POS-tagging (POS: part of speech) off the shelf. I don’t necessarily need much power so I’ve used the small model.

You can just do this:

import spacy
import wikipedia

# settings
wikipedia.set_lang("de")  # set it to use Wikipedia German

lm = spacy.load('de_core_news_sm')  # choose the spaCy language model

page = 'Deutschland'   # title of a Wikipedia page
wikipedia.search(page)  # this will give you best guesses at matching pages

# to get text content
dl = wikipedia.page(page)

# and to create a spaCy document
doc = lm(dl.content)

Initially I thought I’d pull content from some specific long and detailed pages, like the one about Germany itself (with all the history), some film ones, some about the brothers Grimm Märchen (fairytales) and so on. These are all pages rich in text and prone to contain narration, hence verbs. But the counts would be too few to be somewhat representative, and also the set would be biased by whatever set of topics I could think of. So I’ve added a bunch (many) random pages, which you can easily pull with this API:

wikipedia.random(pages=100)   # a 100 random Wikipedia page titles

To extract verbs, following the spaCy POS glossary here and with the from above, I do

for token in doc:
    # take all the verb POS tags
    if token.pos_ in ['AUX', 'VERB', 'MD', 'VB', 
                      'VBD', 'VBG', 'VBN', 'VBP', 'VBZ', 
                      'VAFIN', 'VMFIN', 'VVFIN', 
                       'VV']:
        verbs.append(token.lemma_)  # get the lemma, it'll be the infinitive of the verb
verbs = [v.lower() for v in verbs] # lower case as some will have a capitalised start

I’ve done a few rounds with random pages, considering that not all pages are rich in text (in fact, many are short, there’s also stubs, pages linking to others etc). After a while I ended up with

• content from 3712 pages
• which sums up to 2899268 tokens (words)
• of which I have 10351 unique verbs

That’s very different from my total 336 verbs. It’s been interesting to notice how the count of unique verbs grows quite slowly, it hadn’t saturated yet when I stopped (and it’s another question as to when it would do, maybe deserves another post) but it certainly decelerates. I end up with this (I’m not hand-drawing it this time):

S is not dominating anymore although it’s still quite high, and in fact (this was unexpected for me), A is the top one! There’s a few prefixes starting with A so maybe that’s why, and maybe I haven’t encountered - or learned - those verbs yet. My K seems overinflated, maybe it’s the common verbs that are particularly concentrated there. For the rest, it’s arguably not that different.

As a note: spaCy does mistakes of course so there’s some things in there that aren’t verbs. When isolating the leading letters I also made sure to remove anything that wasn’t alphabetic, wasn’t a Latin character or was a punctuation sign, but some non-verbs still remain.

Well, this was a quite fun exercise and as I said I may dig some more, any comments please tell me here on Bluesky.

I also have a funky newsletter where I share these kind of things, you can check it out and subscribe here if you want.

Books

War and Peace, L Tolstoy

This was the year I partook to a year-long readalong of War and Peace. It was hosted by Simon Haisell on Substack, which you can join for 2025. The book is long but the chapters are short (it was published in daily instalments in the first place), so it’s a wonderful idea for a yearly project. Simon writes a nice daily overview sparking comments and feedback, and participants can have a chat. I really enjoyed the experience and as many said, it’s one of those books that I’d have wanted to read but was put off but the sheer size.

War and Peace draws from themes of human nature, belonging, hypocrisy and truth, but it also contains deep reflections on history and the Russian soul and history. I’ve read the Briggs translation (into English), which is apparently a bit “britishisized”, something I didn’t love. I wish I’d picked up an Italian translation instead, but the other thing about this book is that you can read it over and over again and pick on something else every time, so maybe one day.

Poverty, by America, M. Desmond

Tragic and informative. The tale of how social and economic inequality in the USA came to be a feature, not a bug. You may or may not agree with all suggested causes and the political stance, but it’s nevertheless a great read.

Guns, germs and Steel: the fate of human societies, J Diamond

It’s a cult now, criticised for being simplistic and/or too speculative. Regardless, I’ve learned a lot. Geography and local characteristics played a huge role in making it so that some societies developed faster and more profoundly than others, ending up dominating.

The Patriarchs: the origins of inequality, A Saini

This is great - in fact, I’ve drawn many quotes from it. How one gender ended up giving itself the right to oppress the other didn’t have to be a necessary development, nor it was always and everywhere the case.

Animal Farm, G Orwell

Another classic of course. Doesn’t age.

Movies

Conclave

So good. Power, centuries-old tradition, patriarchy and a lot of twists with some hilarious moments. Clever too.

One Life

The real story of a British young man who saved many Jewish kids from Central Europe organising the Kindertransport - trains that would bring them to the UK in a scheme that allowed them to be temporarily hosted by British families. Many stayed.

Small things like these

Patriarchy and religion in rural Ireland, the ’80s. Inspired by the real events of the Magdalene Laundries. Cillian Murphy is amazing and this, despite being a bit slow, is really good.

Shows

Fallout

It’s on Prime. You don’t have to know about the game before watching it. Dystopian, disturbing. A critic of excessively unregulated capitalism, because we humans can’t moderate ourselves?

Hornblower

This is an old (‘90s) British show, about the naval battles of the Napoleonic period. I like period dramas but I was surprised at how gripping I found this.

Young Sheldon

There was something light I watched too! This is really good, I’ve been late to the party as I felt like it would be hard to replicate TBBT but this, despite being very different in style, is very funny, lighthearted and just enjoyable.

Tools

Readwise & Reader

A year after I signed up, still very happy with it. Good for bookmarking articles, sign up to newsletters, annotate.

Buttondown

Ever since I moved from Substack, very happy. Smooth UI, easy customisation, and most importantly, focus on the writing.

Here’s to another year of new great stuff.

I don’t normally find movie recommenders valuable because of a variety of reasons:

• I want to make sure a movie is highly rated before even considering it for a watch, Rotten Tomatoes is still my to-go source - which means manual checks
• Usually, I want a movie that’s somewhat similar to a specific one I have in mind, depending on mood and feel. I’m not normally interested in lists of movies recommended to me because of an overall estimation based on everything I rated at once. Sometimes I’m in the mood for another ’50s comedy, sometimes for a new history film or a biopic, other times I just want a new stunning nature documentary.

This means it takes me time before deciding what to watch and if I could cut that time it’d be awesome.

That fast.ai module uses a subset of Movielens data as a toy dataset to illustrate how to build a recommender, from scratch as well as using fast.ai utilities. I’ll do just the same but will use the latest edition of the Movielens full data dump and curate it to my needs (more on this below), looking at how results can help me with the above goal. Collaborative filtering is a methodology that essentially matches you with new items based on your ratings on items as well as the ratings of many other users - it learns your preferences and who’s similar to you.

I’ll run a quick model to get a sense of what I end up with, I will favour speed over quality, so long as I end up with something decent, and leave potential improvements to another day.

Movielens data

Movielens is a University of Minnesota project. It is a movie recommender based on collaborative filtering whose datasets are regularly released freely. It exists since the late ’90s, which means its data is the longest-standing one in this area and for this reason it is used for research and to teach ML concepts.

Downloading the data

You can download the latest full version of Movielens movie-ratings dataset from here. At the time of writing, it contains data up until mid-2023.

import requests
import zipfile
import io

MOVIELENS_LATEST_URL = "https://files.grouplens.org/datasets/movielens/ml-latest.zip"

response = requests.get(MOVIELENS_LATEST_URL)
response.status_code  # just to make sure we're getting a 200

# Unzip to local folder
with zipfile.ZipFile(io.BytesIO(response.content)) as zfile:
  zfile.extractall('.')   # will create a local folder ml-latest

This will download a bunch of CSVs with movies info (tags for genre), ratings-movies mappings (what I care about) and some other ones. Metadata is taken from The Movie Database.

Note: I won’t report the whole code here but you can follow this notebook: Colab (view only, I may change things in time), Github (versioned).

Basic EDA

There are ~33.8M ratings in the set, over ~83k movies, from ~330k unique users. The data, as of the README, contains ratings from the start of 1995 to July 2023.

It is out of scope to look at whether peaks correspond to a larger number of movies released, but it’s nevertheless interesting to see that Movielens is still very much used by raters.

“The Shawshank redemption” (released in 1994) has more than 122k ratings - obviously there’s inverse recency bias, in that older films will have been rated more simply because they’ve been around longer.

Curating the set

For reasons mostly related to being able to run a model quickly and ideally without the need for a GPU, I do some cleansing/curation as the data is quite large:

1. remove movies with an avg score below 3.5, because I’m only interested in movies that are generally considered good. Of course, this will remove a lot and will bias the set;
2. remove movies with less than 1000 ratings, to exclude less reliable data - there’s plenty of movies with a lot of ratings so I can afford this. I want to use the results as a filter so I’d rather lose potential recommendations than have bad ones;
3. remove all ratings by users who rated less than 500 movies in total - this is to favour opinions from more committed raters who are power users of Movielens. It will bias the set because I may end up with a userbase that’s less representative of the general population, but I feel that’s fine, I’d rather have qualified opinions (the idea is that people who rate a lot are more likely to rate honestly).

Rating scores in Movielens range from 1 to 5.

I end up with 2145 movies with ~2.1M total ratings from ~3k users. These movies-ratings may still a bit too many for a quick simple model though (read: I tried the below and it was very slow), so I decided to sample them in such a way to preserve the fraction of ratings in each score, taking 10% of all ratings per movie.

Collaborative filtering

I’ve then used collab_learner from fast.ai in the non-neural net modality, which performs gradient descent to learn embeddings for movies and users in the forms of weight factors and biases over a latent space of chosen dimensionality. Embeddings for movies in this latent space encode features of the movies like their vibe, subgenres, quality. They are learned from the users ratings only and not informed by any metadata, which is the fascinating part of collaborative filtering. I won’t go into the details of this but again the fast.ai lesson is great.

With df being our dataframe of movie-ratings (the matrix), I run

# build dataloaders
dls = CollabDataLoaders.from_df(df, 
                                user_name='userId', 
                                item_name='title',
                                rating_name='rating',
                                valid_pct=0.2,
                                bs=64)

# run a collab_learner (EmbeddingDotBias model) with 50 latent factors
learn = collab_learner(dls, n_factors=50, y_range=(0, 5.5), use_nn=False)   # 5.5 because 5 is max score in the data

# and fit it
learn.fit_one_cycle(5, 5e-3, wd=0.1)

The model learns decently, doesn’t overfit and is relatively quick: I can run an epoch in about 30s on CPU (I’ve been using Google Colab). I run it for 5 epochs which gives me decent stats.

The bias for a movie encodes information about how much that movie is generally liked; the bias for users encodes how much that user generally appreciates movies, so these are important terms to use for a general assessment of what I’ve got.

Results

I get these as the movies with the highest movie bias, that is, the most generally appreciated ones:

and these are the one with the least appreciation (remember, these are still good movies overall because of how I curated the dataset):

This sounds credible enough. Now I can use cosine similarity between movie embeddings to retrieve movies most similar to a given one.

For “Mary Poppins” (the 1964 original), I get (asking for 5):

- 'Sleeping Beauty (1959)',
- 'Lady and the Tramp (1955)'
- 'Cinderella (1950)'
- 'King and I, The (1956)'
- 'Beauty and the Beast (1991)'

which are all old animation/family ones, looks good.

For “Paddington” (2014, the first):

 - 'Paddington 2 (2017)',
 - 'Kubo and the Two Strings (2016)'
 - "Dr. Horrible's Sing-Along Blog (2008)"
 - 'Dope (2015)'
 - 'Bo Burnham: Inside (2021)'

it’s good it’s pulling the sequel - these should all be family-friendly except possibly the last one, but the model may have picked on the fun/quirky element and/or it’s just not always good.

For “Hidden Figures” (one of my favourite movies of all time):

 - 'Best Exotic Marigold Hotel, The (2011)'
 - 'Hacksaw Ridge (2016)',
 - 'Sully (2016)'
 - 'Impossible, The (Imposible, Lo) (2012)'
 - 'Saving Mr. Banks (2013)'

this is interesting: the first seems to be a comedy, the fourth is a disaster movie (not a fit, I’d say), the rest are biopics/historical ones which seem like a decent fit.

I could definitely improve the model (see below) but this is already a very good filtering system I can use as-is whenever I’m in the mood for something that would be hard to articulate in a search! Of course, these days I could use chatGPT or the like for this purpose and chances are I’d get decent recommendations, but where would be the fun in that?

Using PCA

Principal Component Analysis is an algebraic technique to reduce dimensionality of large matrices while preserving most of the information and variability. It can be very useful even just to visualise data, like we’ll do here. This part is also piggy-backed from the fast.ai lesson. I have vectors of movies embeddings in the latent space of the 50 dimensions I’ve asked for, but if I PCA them to then draw the 2 components with highest variance I get this plot, which is a good proxy to visually see similarity “clusters”:

The clustering isn’t amazing but you can see some patterns. For instance, the model seems to have picked that “The big Lebowski” and “Fargo” (which makes a lot of references to the former) are similar; there is a Sci-fi cults area (with occasional infiltrates like “Die Hard”); there’s an area of quirky crime ones and so on.

Conclusions and things I could improve

It wasn’t my aim to create and tweak a great model, but I’ve (quickly, thanks to fast.ai!) got my filter which I can now use to have a list of movies similar to one I like with no effort. The movies are all highly rated by the Movielens community, which doesn’t necessarily mean they’d be highly rated on Rotten Tomatoes, but I’m sure the correlation will be high (it will be interesting to find cases where this isn’t true). Every time Movielens produces a new data dump I can update my model.

A lot of things could be improved:

• first of all, reduce the draconian size-reduction operations on the dataset I had to do to keep this fast in training
• train on GPU - I didn’t invest time or money into this and Google Colab isn’t reliable in the free tier because one moment you have a GPU and the next you don’t anymore, but with a little expenditure you can train more properly. Or, you can use the Kaggle free GPUs too.
• tie the results to Rotten Tomatoes (doesn’t have an API AFAIK though, so you’d have to scrape) as a further post-processing filter

Hope you enjoyed this, chat to me on Bluesky.

“Now, more often than not, I find myself explaining to colleagues, to bosses, to investors, and to customers that the AI actually is wrong even while it sounds right.”

This is an interesting situation indeed. Working in data science/Machine Learning has always been a fun ride where rapid change is the norm and the best skill is adaptation, but generative AI has partially flipped the very nature of some workflows.

If you work with LLMs/VLMs you are likely in a situation where:

1. building some things is easier and quicker
2. testing results for quality is shakier
3. convincing others that what you built is valuable is much easier (that’s not a good thing)

Building some things is easier

Some classes of problems and some instances of those classes can now be well solved with an LLM/VLM. Trivial examples are classifying a piece of text or image (it won’t necessarily apply to all use cases though). If you’re devising a cookery app and need to group recipes into cuisines, chances are an LLM can do that for you very well. I’ve asked ChatGPT to extract text from this BBC Good Food recipe, create a Markdown for it and then tell me what kind of cuisine it is, the response was

“The “Pasta with Salmon & Peas” recipe combines elements from both Italian and British cuisines. The use of pasta and crème fraîche reflects Italian culinary traditions, while the inclusion of salmon and peas is common in British cooking. Therefore, this dish can be considered a fusion of Italian and British cuisines.”

which, though I’m not sure about the crème fraîche part, I don’t overall disagree with.

However, I’m not sure how to call it (conventional?), but non-GenAI-based ML still has a lot to offer - in fact, I bet most of the ML around the business world is still linear regressions and random forests. Not to mention it’s often the best for your buck: the LLM fauna now has many cheap-enough options and the general thinking is that prices of established models will keep decreasing because that’s how basic economics works. But there’ll always be shinier, more capable models that are expensive due to the need to recover research and training spend. Also, LLMs can’t help with everything.

Testing quality is shakier

This is my main point of nervousness with LLMs. You usually test their behaviour on a bunch of data. If you’ve been careful, you selected the data to be representative of your domain, stratified, large enough and unbiased. You make this dataset into an eval set which you run periodically to check for robustness. But the fact that results may look good doesn’t tell you how the LLM will behave at scale. Hallucinations, gibberish and garbage may arise, and there’s always the risk of discriminatory language, something to be very wary of especially when building tooling that impacts on people’s lives. For some things, there’s ways to cope.

Confirmation bias looms above us humans in every circumstance but I wouldn’t be surprised to learn that we are easier to fall victim to it when LLMs are involved. It can be easier to see goodness when responses are in the form of human speech rather than categories and numbers.

Further, similarly to what happens with clinical trials for drugs, side effects may be rare enough to only appear when the product is out in the population at large. Things like these are not exclusive to LLMs, but the chance for non-reproducibility is higher. The only way¹ is to monitor, build alerts, trace and fix when possible.

It doesn’t help that for closed LLM models we don’t know the details of their structure. The scientific process is in part violated: you don’t control the tool, you can’t inspect it, you can only see what it does ex-post and infer what it is capable of.

Convincing “stakeholders” is much easier

Model interpretability has always been an important part of M/stats work. In fact, it’s a field of its own. Back in the pre-LLM days, when you as a data/ML scientist built an algorithm, other people (the “stakeholders”) may have had a skeptical attitude towards it unless they had a sense of what determined its results. They wanted to know why the churn model predicted Mr. Smith will cancel the subscription in 3 months, or what was the driver behind that forecasted growth rate of app downloads next quarter. Often, their skepticism was well meant and actually useful to the data/ML team, as a large part of building good ML tooling is being able to communicate their value. It is hard to trust a system which gives you back arid numbers and nothing else. The job of the data team is to build something good enough that is wrong a small fraction of the time and that, ideally, when it’s wrong it doesn’t screw things up, and to communicate this in a friendly, factual and metrics-oriented way.

Fast-forward to now and the same people, as well as (almost) everyone else, seem to just trust LLMs. I think there’s two reasons for this:

1. everyone can interrogate LLMs themselves, non-tech teams don’t feel at the mercy of the data/ML team anymore and this is empowering;
2. LLMs talk to you in natural, polite and human-like ways - after all, who wouldn’t trust a competent orator?

It is hard to distrust a system that gives you back nicely packaged natural-language explanations for its choices.

The data/ML team now has the added job of instilling a healthy dose of sane lost-skepticism back so that everyone can do better than accepting things at face value.

The script-flipping means we should now be even more focused on extensive testing, even more alert about possible bad output, even more solid in our skepticism-bearing and question-asking.