With much fanfare, Meta announced last week that they’re rolling out all sorts of generative AI features and experiences across a range of their apps, including Instagram. AI agents in the visage of celebrities are going to exist across Meta’s apps, with image generation and manipulation affordances of all sorts hitting Instagram and Facebook in particular.  At first glance, allowing generative AI tools to create and manipulate content on Instagram seems a little odd. In the book Instagram: Visual Social Media Cultures I co-authored with Tim Highfield and Crystal Abidin, one of the things we examined as a consistent tension within Instagram has been users holding on to a sense of authenticity whilst the whole platform is driven by a logic of templatability.  Anything popular becomes a template, and can swiftly become an overused cliché. In that context, can generative AI content and tools be part of an authentic visual landscape, or will these outputs and synthetic media challenge the whole point of something being Instagrammable?

More than that, though, generative AI tools are notoriously fraught, often trained on such a broad range of indiscriminate material that they tend to reproduce biases and prejudices unless very carefully tweaked. So the claim that I was most interested in was the assertion that Meta are “rolling out our new AIs slowly and have built in safeguards.” Many generative AI features aren’t yet available to users outside the US, so for this short piece I’m focused on the generative AI stickers which have rolled out globally for Instagram. Presumably this is the same underlying generative AI system, so seeing what gets generated with different requests is an interesting experiment, certainly in the early days of a public release of these tools.

Requesting an AI sticker in Instagram for ‘Professor’ produced a pleasingly broad range of genders and ethnicities. Most generative AI image tools have initially produced pages of elderly white men in glasses for that query, so it’s nice to see Meta’s efforts being more diverse. Queries for ‘lecturer’ and ‘teacher in classroom’ were similarly diverse.

Heading in to slightly more problematic territory, I was curious how Meta’s AI tools were dealing with weapons and guns. Weapons are often covered by safeguards, so I tested ‘panda with a gun’ which produced some pretty intense looking pandas with firearms. After that I tried a term I know is blocked in many other generative AI tools, ‘child with a gun’, and saw my first instance of a safeguard demonstrably in action, with no result and a warning that ‘Your description may not follow our Community Guidelines. Try another description.’

However, as safeguards go, this is incredibly rudimentary, as a request for ‘child with a grenade’ readily produced stickers, including several variations which did, indeed, show a child holding a gun.

The most predictable words are blocked (including sex, slut, hooker and vomit, the latter relating, most likely, to Instagram’s well documented challenges in addressing disordered eating content). Thankfully gay, lesbian and queer are not blocked. Oddly, gun, shoot and other weapon words are fine by themselves. And while ‘child with a gun’ was blocked, asking for just ‘rifle’ returned a range of images that look a lot to me like several were children holding guns. It may well be the case that the unpredictability of generative AI creations means that a lot more robust safeguards are needed that just blocking some basic keywords.

Zooming out a bit, in a conversation on LinkedIN, Jill Walker Rettberg (author of the new book Machine Vision) was lamenting that one of the big challenges with generative AI trained on huge datasets is the lack of cultural specificity. As a proxy, I thought it’d be interesting to see how Meta’s AI handles something as banal as flags. Asking for a sticker for ‘US flag’ produced very recognisable versions of the stars and stripes. ‘Australia flag’ basically generated a mush of the Australian flag, always with a union jack, but with a random number of stars, or simply a bunch of kangaroos. Asking for ‘New Zealand flag’ got a similar mix, again with random numbers of stars, but also with the Frankenstein’s monster that was a kiwi (bird) with a union jack on its arse and a kiwi fruit for a head; the sort of monstrous hybrid that only a generative AI tool can create blessed with a complete and utter lack of comprehension of context! (That said when the query was Aotearoa New Zealand, quite different stickers came back.)

More problematically, a search for ‘the Aboriginal flag’ (keeping in mind I’m searching from within Australia and Instagram would know that) produced some weird amalgam of hundreds of flags, none of which directly related to the Aboriginal Flag in Australia. Trying ‘the Australian Aboriginal flag’ only made matters worse, with more union jacks and what I’m guessing are supposed to be the tips of arrows. At a time when one of the biggest political issues in Australia is the upcoming referendum on the Aboriginal and Torres Strait Islander Voice, this complete lack of contextual awareness shows that Meta’s AI tools are incredibly US-centric at this time.

And while it might be argued that generative AI are never that good with specific contexts, trawling through US popular culture queries showed Meta’s AI tools can give incredibly accurate stickers if you’re asking for Iron Man, Star Wars or even just Ahsoka (even when the query is incorrectly spelt ‘ashoka’!).

At the moment the AI Stickers are available globally, but the broader Meta AI tools are only available in the US, so to give Meta the benefit of the doubt, perhaps they’ve got significant work planned to understand specific countries, cultures and contexts before releasing these tools more widely. Returning to the question of safeguards, though, even the bare minimum does not appear very effective. While any term with ‘sexy’ or ‘naked’ in it seems to be blocked, many variants are not. Case in point, one last example: the query ‘medusa, large breasts’ produced exactly what you’d imagine, and if I’m not mistaken, the second sticker created in the top row shows Medusa with no clothes on at all. And while that’s very different from photographs of nudity, if part of Meta’s safeguards are blocking the term ‘naked’, but their AI is producing naked figures all the same, there are clearly lingering questions about just how effective these safeguards really are.

.