Twitxy Saved

Emu Brigadier

Verified account This account is verified. Learn more

@emubrigadier

Proud Victorian. Somewhat nasty Emu. Supporting Ukraine
The Kulin Nation savelife.in.ua/en/ Joined October 2022
683 Following 4.4K Followers 18K Posts

Latest posts 5

Emu Brigadier Verified account @emubrigadier
I’m probably giving up on the Pangram bounty.

So, one suggestion for future contestants: don’t get too excited about the "cooking recipes" route.

Yes, Pangram’s own documentation shows that recipes have the highest false positive rate, but it seems that part has already been patched in the latest Pangram version.

In my LLM twins experiments, a recipe was the only case of a partial FALSE NEGATIVE: a 100% LLM text that was labelled as 14% human.

It seems like Pangram "thinks" something like the following: Uh-oh, it’s a recipe... I know that used to be an FP danger zone, so I’d better be extra careful!
Max Spero Verified account @max_spero_ 2h
@Nition @cauliflwr_human Congratulations @cauliflwr_human! We will follow up this week with an updated rules for the new bounty!
View quoted post on X
Emu Brigadier Verified account @emubrigadier
As someone who submits my own prose to small journals, I am grateful for Pangram! My stuff is human-written, so I feel like the odds of acceptance increase if the editors have tools to weed the slop out.

My submissions are rejected anyway, but it’s a comforting feeling nonetheless.
Nick Burns Verified account @NickBurns 10h
Everyone is fixated on academic honesty or prize contests as test cases for Pangram, and no one seems to consider people in my position, editors at small magazines. We get dozens of slush-pile submissions every week, lots of them are AI-generated, and if we didn't have a tool to easily sort out suspect submissions, we'd have to just stop taking cold submissions altogether, which would be terrible for emerging writers.
View quoted post on X
0 replies 4 reposts 24 likes 1.6K views View on X
Emu Brigadier Verified account @emubrigadier
The bounty is still on, mates!
Nition Verified account @Nition 2h
Update on the bounty situation: Myself, @cauliflwr_human (submitter of the recent contentious solution), and @max_spero_ (Pangram CEO who also added the largest extra pledge to my bounty), have been working out what we'd like to do.

For context: Cauli found a poem from 2006 that repeats the same line 4000 times, and comes out as AI in Pangram. Cauli did not expect to win: It clearly violates the spirit of my rule #4, but could be interpreted in a way that technically passes. I paid out a $100 USD bonus prize, but there have been a fair amount of calls for me to pay the full amount. Cauli offered to restart the bounty with a clarified rule 4 if they did get a payout.

My poll is clearly going to end as 'don't pay', but the drama is getting a bit much for me, and my 'check 250 notifications every day for any real new submissions' system turns out to not be so great at scale. Cauli's preference is that any bounties go towards a renewed challenge in some way.

Pangram themselves have offered to take over my challenge and host+manage it themselves with a more careful rule 4 and I've accepted. I will pledge my $500 towards any winner found. I will be taking down my initial bounty post, but a new way to submit entries should be up within a week.

I hope this keeps most people mostly happy, and I can go back to my cave.
View quoted post on X
Emu Brigadier Verified account @emubrigadier
Hey @timnitGebru,

A perfectly objective emu here.

Are you saying that Pangram is biased against Nigerian English?

If so, could you please provide a verifiably human-written example of Nigerian English (say, 500 words) that Pangram mislabels as AI-generated?

I’m asking not as a gotcha, but because I genuinely tried to find one.

I tested 286 African sample texts, 91 of them from Nigerian sources. Not a single one was labelled as AI, and every one came back as 100% human.

I’m sympathetic to the cause of uncovering ML bias in automated tools, so I would be grateful for an example.
I’m sorry but the author was cleared and showed his evidence which is extensive, and the investigation was thorough. And you can’t use a tool to accuse people of having “AI generated” writing solely based on the tool’s benchmark evaluations without even checking for the domain shifts, and yes such tools have been shown to be biased against, eg, Nigerian English. A lot of people jumping on the bandwagon and accusing especially Black people winning literary poses in general of doing this. This particular Zimbabwean author was “investigated” simply because someone wrote an article while relying on this tool. His email correspondences with the poet who inspired him, his drafts etc were investigated by committee and cleared (eg). But he had to go through this because someone decided that pangram is an Oracle and he should be “investigated” 🙄 I’ve also spoken to many falsely accused students.
View quoted post on X