Environmental impact of LLMs

A reader asked: what is the environmental impact of large language models (LLMs)? So I read some articles on the subject, and made comparisons to other technologies, such as video games and video streaming. My conclusion is that the environmental footprint is large enough that we shouldn’t ignore it, but I think people are overreacting.

Pricing

I’m not an expert in assessing environmental impact, but I’ve had a bit of experience assessing computational prices for LLMs. Pricing might be a good proxy for carbon footprint, because it doesn’t just represent energy costs, but also the costs of building and operating a data center. My guess is that across many different kinds of computation tasks, the carbon footprint per dollar spent is roughly similar. And in my experience, LLMs are far from the most significant computational cost in a typical tech company.

[Read more…]

I worked on LLMs

Generative AI sure is the talk of the town these days. Controversial, is it not? It’s with some trepidation that I disclose that I spent the last 6 months becoming an expert in large language models (LLMs). Earlier this year when I moseyed through the foundational LLM paper, that was where it began.

I’d like to start talking about this more, because I’ve been frustrated with the public conversation. Among both anti-AI folks as well as AI enthusiasts, people have weird, impossible expectations from LLMs, while being ignorant of other capabilities. I’d like to provide a reality check, so that readers can be more informed as they argue about it.

[Read more…]

Link Roundup: June 2024

Orientalism: Desert Level Music vs Actual Middle-Eastern Music | Fayra Faraji (video, 1:36 hours) – This video explains how orientalist music has virtually nothing to do with actual music from the Middle East.  The music uses a hodgepodge of different instruments and musical styles that come from vastly different contexts.  They nearly exclusively use the double-harmonic and phrygian mode, not because those are particularly common in Middle-Eastern music, but rather because it’s uncommon in other western music and yet fits within the 12TET system.

As a fan of xenharmonic/microtonal music, I know that many non-western music traditions use different tuning systems–the Maqam traditions are particularly notable.  I appreciate such music as it comes into my awareness, and definitely wish it were more widely distributed.  That said, I’m very aware that I come from a western musical tradition, and the very first thing I hear in microtonal music is a sense of uniqueness relative to my musical context and training.  When I think about non-Western musical traditions, I imagine a whole history and culture where these musical characteristics are just normal, just a medium used to express something else entirely.  That just isn’t my perspective, I cannot hear it that way.

[Read more…]

Origami: Box Head

Box Head

Box Head, designed by Boice

This iconic model is one that we folded at the East Bay Origami Convention back in March.  It takes inspiration from Boice’s Head Empty model, which is a person in a coat and tie, with a cube for a head.  Taking that same idea to the extreme, now we have a little doll with a giant cube for a head.  It’s pretty hard to get a photo of the thing, because from any view from above, the giant head eclipses the rest of the body.  If you could see it from above, you would see that the cube is open on the top and in the back.

Boice has a video tutorial.  It’s easier than it looks, but you may want to start with large paper.  I think it’s a good introduction to the box pleating method.

The tricky part was getting it to stand up.  I chose the fold the feet a bit differently from the instructions, giving it giant duck feet.

Mysteries do not need to be solvable

In the past year I’ve gotten into reading mystery novels, and this has reinforced one of my strongly held opinions about the genre. There is a mistaken preconception about mystery novels, that the reader ought to be able to solve the mystery. This simply is not true. There are some mystery stories that are meant be solvable, but it’s a minority of mystery stories that I’ve seen. Solvability is not the primary appeal of the genre, or at least it’s not the appeal to me.

The reason I know this, is because when I was young, we had a “complete works of Sherlock Holmes” book, which had all the short stories. I didn’t read them all, but I read enough to know that Sherlock Holmes stories were not solvable. Usually, Sherlock Holmes would pull some clue out of thin air, that hadn’t been mentioned before; or else there would be an event that led to the mystery being solved. It was unambiguous that most stories were not even trying to be solvable. The mysteries were trying, first and foremost, to be stories. There’s something the reader doesn’t know (rising tension), and then Sherlock Holmes explains it (releasing tension), and that’s a simple but effective narrative arc.

[Read more…]

The information theory of Mysterium

A question that keeps me up at night is “What is the theoretical best you can do in Mysterium?” I’m exaggerating a bit, but it is a pointless question about a silly board game that I nonetheless spent too long thinking about. I went so far as to watch a series of lectures about information theory–listening to it in the background while in dance class, as one does. I never solved the problem, but let me at least explain what the problem is.

Mysterium

Mysterium is a cooperative board game where the players are trying to solve a murder mystery via psychic communication with the victim. One player takes the role of the ghost, and the rest take the role of psychic mediums. The ghost is not allowed to speak, and may only communicate through cryptic visions. The visions are represented by cards with surreal artwork. For example, one card has two people climbing into a giant fish mouth, another has a tarantula-like thing over a chandelier. After the mediums receive their visions, they discuss what they mean and make their guesses; and the whole time the ghost giggles about how wrong they are.

Example visions: two people climbing into a fish's mouth; a polar bear and spirit owl read a book; a chandelier hanging from strings from a tarantula's mouth

Examples of vision cards.  Source: Mysterium rulebook.

[Read more…]

Link Roundup: May 2024

This month, the Ace Community Survey published a report on sexual violence.

The real story of Gamergate 2 (conspiracy theories, crybullying, and a consumer revolt) | NeverKnowsBest (video, 1:38 hours) – I’ve been loosely following the story of antiwoke backlash currently going on in the gaming community, but it’s pretty hard to take seriously when it’s founded on something so ridiculous and petty.  NeverKnowsBest clearly explains the sequence of events and issues of concern, and persuaded me to take it more seriously.  After all, the Gamergate of a decade ago also started with something silly and petty, but it snowballed into something bigger by galvanizing the alt-right presence within gaming communities.  Also, the media environment is very different from how it was a decade ago, with traditional games journalism being far less influential.

What these neo-gamergaters want is games that cater more to their political tastes, i.e. centering straight white men, dropping black & queer characters, making women sex objects again, etc.  That’s already hard enough to sympathize with, but then they add all these conspiratorial claims involving a tiny consulting company, and ESG investment.  When it comes to progressive politics in games, they can’t accept the more basic explanation that some game devs are pretty progressive, so instead they believe that investors of all people, are the ones pushing the progressive agenda.  This is so obviously wrong, just look at all the indie games, which are less beholden to investors and publishers than ever.

How Does Fiction Affect Reality? | Thing of Things – Ozy discusses the evidence regarding the real world impact of fiction.  There’s surprisingly little, although fiction can impact social norms such as norms around family size.  As a critic, when I criticize a work of fiction, the purpose is rarely to say “this work of fiction is causing harm and should not exist”.  Often, criticism is just an intrinsically fun and valuable activity, in the same way that fiction itself is just an intrinsically fun and valuable activity, independent of whether it has an impact on society.  Yes, there is criticism, such as feminist criticism, that wants to push towards positive social change.  But I think it’s more important for people to engage with criticism than to avoid engaging in the material being criticized.

[Read more…]