I had a spare half an hour this morning, and my newspaper had a list of 15 things the next generation of ChatGPT will be able to do for you. There’s been some discussion about what ChatGPT does on discussion groups and how convincing it is. Someone, for example, asked it about clipping counters in Advanced Squad Leader and it gave reasonable arguments both in favour and against the idea. There’s suggestions that it is just a really smart search engine with a text generator attached. 

I thought it would be interesting to see if it could produce anything remotely serviceable for the historical notes for my upcoming set of War of Spanish Succession rules, so I asked it to describe infantry tactics during the War of Spanish Succession.

It started off all right. It got the dates right, and then talked about linear firepower tactics. It then went on to talk about the use of columns and skirmishers. I replied that this sounded like Napoleonic warfare, and it agreed with me, and modified its response (Changed its mind, perhaps?) to conform more with accepted understanding. So, it appeared to be learning. Or is it like that bloke in the pub who knows everything, but changes his mind to agree with you when caught out?

When pushed on the differences between French and British tactics, it then backslid dramatically and started talking about columns and voltigeurs. On being challenged it accepted it was wrong, and gave me an answer that suggested the French liked bayonets („that make a musket like a type of spear“) and charging, and the British liked firing.

Where it got interesting was when I asked about Dutch tactics. It started off okay, with a discussion of linear tactics and firepower. Then it got weird, and asserted that the Dutch used native troops from their colonies, armed with spears and bows and arrows, to attack enemy supply routes. I asked it to clarify if this was in Europe, and it assured me that it was, and that they were deployed in both France and Germany. It said they were known as the „Black Javanese“. I pressed my little AI friend further. When were they deployed, I asked. The response was this:

Now that is REALLY specific. It almost made me doubt myself. I took the bull by the horns, and went straight in with the „I think you are mistaken“ approach. Sure enough, it finally backed down, and admitted that „it appears that there is little to no historical evidence that Javanese troops were deployed to Europe during the War of Spanish Succession“.

It took some pressure to get it to admit it had made a mistake, but we got there in the end but only because I knew it was wrong and asked the right questions. Reading what it wrote was very plausible. I did some follow up on Austrian tactics and the use of battalion guns, which was mostly convincing. I’ve had problems tracking down the exact usage of battalion guns in the WSS, so I’m not sure if the answer is completely reliable. I consequently won’t be using any of its „research“.

I moved on and asked it about the date of the battle of Edgcote. It went for the 26th July 1469. We had an argument. It insisted that the 26th July 1469 was a Monday (it wasn’t – it was a Wednesday). It wasn’t until I rubbed its metaphorical nose in the Coventry Leet Book that it finally caved and accepted the 24th. Where ever it is getting its information, it isn’t from Wikipedia.

Where it did get uncomfortable was around the use of „Edgecote Moor“ as the name of the battle. It insisted the earliest usage was in George Baker’s History of Northampton written in 1822. It isn’t, so I pressed it further. It revised its view and said it was in Hall’s Chronicle in the 16th century. No it isn’t. He refers to it as „Heggecote“. When I told it I thought the earliest usage was in 1848, it agreed, and said that Baker’s book was actually written in 1848. No it wasn’t. And so it goes on. Like the pub bore, caught out in a lie, it changes its story repeatedly, mixing its facts up, and apparently making stuff up. It even quoted from a book it said was written in 1995, quoting title and author. The book doesn’t exist.

So we have a machine that writes plausibly using an algorithm that seems to make stuff up, whether by design or accident. I sure won’t be using it to submit any of my homework assignments.

On the other hand, it is making a decent fist of writing a set of wargame rules:

We are currently looking at each phase in turn. I have narrowed the parameters to 15mm figures on a 6′ x 4′ table, using 15mm figures. So far we have movement rates, and are working on the combat rules.

Watch this space. I will be making the final set available for download.