When Safety Forgot about Scammers

For a while, I’ve been thinking about how the online dating moderation space shifted from focusing primarily on tackling masses of spam, much of which was scam-related, to prioritizing individual safety. In theory, that should’ve been a good thing – but in reality, something important got lost along the way. I feel somewhat responsible for that because I led this major shift, but I did try to rebalance this a few years later. Even so, scams are still not taken nearly as seriously as user safety on dating sites any more.

Back in the early days of modern dating sites, scam prevention wasn’t much of a blip on the radar. In 2013, when I was head of moderation at OkCupid, we mostly cared about the spam that always precedes scams because dating site users hated seeing obvious spam. A site full of spammers is going to lose users, many of whom are convinced (wrongly in most cases) that they are created by the site to increase the number of users people see. An abundance of spammy profiles impacted our user base, and although we didn’t charge money at the time, so there was no bottom line to worry about yet, we still wanted to keep people happy and on the site, and for them to rave about us to their friends.

The workflow and tools ultimately led us to prioritize spam as a whole, rather than concentrating just on scams. Little thought was ever given to the consequences of the spam-to-scam path anywhere on the Internet. The team responsible for handling potential scammers was called the “spammer admins”, and the tool used by almost all front-line moderators in the company was a spam reduction tool called Spamadmin. That tells you everything about priorities at the time.

As time went on, the moderation teams became full-time, less informal, and better trained and managed. My focus completely shifted from spam and general abuse toward individual safety (harassment, stalking, and assault prevention). We still had a team dealing with spammers, but they were mostly made up of newer workers, with most of the experienced ones moving up through the safety and incident-response teams.

When Elie Seidman joined as the CEO of OkCupid in 2016, I was changing the way I thought of safety to really concentrate on dealing with individuals who may cause actual harm to our users or dealing with ones who already had. We worked on creating teams with people who could talk to victims of crimes and often help them with the cold system they would face if they reported anything. The focus of our safety teams changed dramatically, yet scam victims still received no support. Elie sat through lots of late-night chats with me about my visions for safety and he saw first-hand how successful my team was at what it did when our investigations team caught the person sending him anthrax and other nasty things through the post and passed the information to the FBI leading to his arrest and conviction. This was all happening at the time OkCupid was in a major lawsuit brought by a rape victim in Georgia who said that we didn’t do enough to protect her against a known abuser coming back to the site in 2014 which at least changed a lot in how we dealt with serious incident reports.

When Elie left to go to Tinder in 2018, he promoted me to Head of Safety at OkCupid and took my Trust and Safety ethos, with its distilled focus of prioritising in-person abusers with him. At the same time, the rest of the dating industry was also moving further toward prioritizing in-person safety – with Bumble making big PR wins by announcing innovative safety features that OkCupid had been doing quietly for years. By the time Match Group created a new centralised Safety Team – staffed with external trophy hires who had no experience with dating sites – any focus on scams had almost entirely disappeared.

A couple of years after Elie moved to Tinder in 2018, Match Group hired a new central Safety Team of ‘trophy hires’ that was exclusively focused on this shift in safety and had zero interest in scams. I had been one of the key people changing the conversation from scams toward individual safety, and I was starting to rethink and regret that. I tried to bring scams back into the discussion, but it was too late to do too much by then.

Meanwhile, scams weren’t slowing down. Match Group has a group-wide mentor program, and in early 2021 my mentee at Tinder was scammed out of $60,000-80,000 in a very clever crypto scam. There was no internal training or discussion about scams like this in a company that theoretically has some of the best moderation and user safety in the world. In the same month, on the opposite side of the USA, an OkCupid executive’s mother-in-law lost $300,000 in what looked to be the same scam. We compared messages they both received, and they were nearly identical. These weren’t just “small-time” cons – this was organised crime operating at a massive scale.

Around the same time, a surge in sextortion scams targeted Muslim men, preying on cultural vulnerabilities. Scammers posing as women or gay men would lure them into explicit conversations that they video captured and then threaten to expose them to their families, communities, and workplaces. The emotional blackmail was brutal, and the consequences, in many cases, were life-ending.

Back in 2018, my focus was almost entirely on in-person safety – but that year, something shifted. Scams came back into focus for me after reading Will Ferguson’s 2012 novel “419”, about notorious Nigerian internet scams and the crime rings behind them. The book focuses on the family of a victim who took her own life after falling for a romance scam, and her daughter who tries to find out the truth. The book was very well researched and became required reading for my teams. I started to refocus on the fact that scams are user safety, with potential victims facing financial ruin, blackmail, and unfortunately, often suicides. These aren’t just “spam” issues, they’re absolutely safety issues. But the industry spent years treating them as separate problems, forcing a staffing effort choice between protecting people from bodily harm or protecting them from financial devastation. This divide was a mistake.

It’s cold, but the truth is that while sexual assault is devastating, a scam can also destroy an entire extended family in many different ways – and often does. If someone takes their own life because of a scam, the ripple effects are enormous. The fact that the industry ever framed this as an “either/or” issue instead of recognising the catastrophic damage both can cause was a fundamental (but understandable) failure. That mindset must change.

When Match Group safety started taking more notice of Pairs, a successful Japanese dating site it had bought a few years before, I started talking to Tomomi Tanaka there about scams because she’d heard me trying to get some effort put into them. Scams were somewhat alien to Japanese culture, so users of Pairs were fairly easy targets. By 2021, I had spent three years trying to bring Romance Scams, Sextortion, and Crypto Scams into the central safety conversation – all without success, even after an executive’s family had been scammed. The new central team was not interested and was resistant to anything they didn’t know or understand. In 2021, Tomomi created a report titled Cryptocurrency Scammers in Japan. I had hoped it would gain traction, but in my time there, it didn’t. By that point, I was ready to leave anyway.

[ I also published this one on my personal blog ]

Posted in Groupthink, Ponderings, Safety | Leave a comment

Practical Lessons in Humility

Back in 1987, a colleague and I were using the large computer systems of the University of Leicester to run rather a lot of number crunching. Leicester had a large, well connected and well managed VAX Cluster which was ideal as a platform to collect password files and analyse them for erm… Statistics. It’s probably fair to say that we didn’t have permission to be using the Leicester systems but life was somewhat different then.

We’d been using this machine for a few months and one day we spotted that something was amiss. The Systems Manager of the machine had activated process accounting which effectively meant that everything we had done had been logged and there was a complete audit-trail of our naughtiness.

This was a long time ago, in the days before writeable CDs but we had an equally bad adversary in the guise of the system console of the VAX 8600 we were squatting on. In those days consoles looked like this:

A yellowing (originally cream) typewriter console with 80-column fanfold printer paper in it.A proper one would have full-width paper but it was hard enough finding this image :)

LA120 VMS System Console

We had a problem at this point, in that we didn’t really want the people at Leicester knowing what we’d been doing and at the same time we were somewhat stuck because it had all been printed out and you can’t remotely edit a teletypewriter trail. Although we’d turned off the accounting, and disabled any future accounting, we were in a little bit of a hole.

We pondered this for a while and came up with a nice and neat solution – We’d get the logs from the last week and we’d edit any traces of ourselves out – We’d then print this to the printer so that anybody looking back over it for however many days wouldn’t see us. A little bit of editing and a couple of hundred pages of printing later, we’d wiped all traces of our activity out. Taking the hint that we probably weren’t wanted, we politely left, breathing a small sigh of relief that we’d narrowly escaped detection.

A couple of years later, I was doing my postgraduate at Leicester University and I went for a drink with their Systems Manager, Pete Humble – I was a little more well known at this point as one of the “good guys” and we got to chatting about the old days and for some reason the subject of tracking a big hack a couple of years back came up. Pete told the story of how some people had been using his machines for nefarious activities but they’d always managed to evade any logging so nobody had a clue what they were doing.

Apparently he came up with the idea of setting process accounting on the miscreants and logging this all to the console. I grinned knowingly and smugly asked him how that went for him and was somewhat taken aback when he explained that those pesky kids had cleverly reprinted a few of days of sanitised logs to hide themselves BUT, that every night, he had marked the stack of paper on the edge with a thick red marker and when he saw that the evening had produced an extra foot or so of printout above the last red mark, he’d realised what had happened.

At that point, I bought the drinks for the rest of the evening and congratulated him on being the only person to have ever caught me, with something as apparently low tech as a red permanent marker pen.

Peet remained one of my best friends until he died a few years ago, and I like to think that he taught me a lot more than not to be too smug in underestimating lower-tech solutions.

Posted in Amusement, Narrative | 1 Comment

Thank you Jeanette

Just as an amused aside for a Saturday afternoon – Back in 1998 I was building what was then Cellnet’s Genie System, it is now O2 but you can’t blame me for any of that, Genie was actually very good for its time.

I was bored one day and since Cellnet didn’t have anyone in charge of sex and pornography like BT then did, I couldn’t pass the buck on the username registration issue so I decided to do it myself. I sat down and made a list of rude words, I went round the office and asked everyone their favourite rude words, I polled IRC for various rude words and in the end, I had a pretty comprehensive list of rude words (which before you ask, I have now lost).

I decided to ignore the usual “Scunthorpe” issues, and just blanket ban registrations with any of them in – It wouldn’t tell them that they couldn’t use that name, it just told them that the username already existed.

I’d forgotten all about this about 6 months later until I got a call from the Cellnet support people up in Thurso – They had an issue with a user who couldn’t register and they’d tried to do it for her and still they couldn’t. They’d tried adding a 2, a 3 all the way up to 100 to her name and still, they kept getting told the user already existed. Nobody had been able to solve the problem, the poor user was upset that she had so many namesakes and I was just on the phone giggling which didn’t help at all.

I did fix it for her, as a one off, but to this day, the memory of poor old Jeanette Quimby still brings a smile to my face.

Posted in Amusement, Narrative | 1 Comment

Learning from History

I first left the Commercial Security industry back in 1999 when I decided to create Recruitment.com. After I moved on from there, I ended up being interviewed for a directorship of a company making Intrusion Detection Systems.

They asked me a question which surprised me – It shouldn’t have but it did – “You haven’t done anything in the industry for over four years now. What makes you think that your skills are relevant?”

Maybe this is a problem with the whole concept of Commercial Security or indeed anyone who takes a wide view of security as interconnecting social and technological parts – I don’t think people know what it is. I think they are looking for ongoing training courses and certifications that give you letters after your name pronouncing that some random software company considers you to be an expert in something. I don’t have any of these and really I don’t want them. On the engineering side I am not going to mend a Cisco router myself and on the policy side, the various qualifications seem utterly meaningless and at odds with one another. The people who create these certifications are trying to push a methodology and singular methodologies are not always a good way to go.

I explained politely that very little that I do has changed much in the last few hundred years – Security is security, always has been, always will be. The methods of delivery differ but the concepts don’t. You can learn more about my type of security from sitting in the middle of a large castle and pondering why they built it the way they did. The inhabitants of the castle needed to be able to come and go and get about their daily lives, there were markets, shops and whole villages inside the larger castles and these communities needed protecting from outsiders and insiders alike. There’s nothing really new conceptually. Take the Trojan Horse – This was first deployed in name over 3,000 years ago and even then one of Troy’s very experienced voices was ignored when he said they should burn it. A fine example of management-override in action which can still teach us more real-world lessons than countless courses and certificates.

It is in the commercial interests of various corporations to have us believe that they can create expert security people by giving them a training course and a certification. They can’t, no more than military basic training can create a good soldier. Sandhurst and Westpoint teach the classics to the military officers of the future for good reasons – A firm grounding in history, an appreciation that you can learn from the past and an understanding that the last 4,000 years of human development were not just a waste of time; these are the tools that somebody needs to become a good security all-rounder. Then they need the experience of applying it, experience of where it goes wrong and experience of learning to live with the fact that most of the time, nobody will listen to them. Don’t worry – History will also teach that this was always the case as Laocoön would have pointed out if he hadn’t been silenced permanently.

Posted in Narrative, Rants | Tagged , , , , , | 2 Comments

Who will buy, our wonderful snake-oil?

“I am patient with stupidity but not with those who are proud of it.” — Edith Sitwell

In my previous job; which was online-moderation, I did some technical-support for a while. I didn’t last long because their naivety and lack of understanding of their basic profession simply astonished me. One of the things we had a huge difference of opinion about was trust. In my mind and experience, if you are going to trust somebody to login to a client’s accounts with the power of a moderator then you have to have some basic trust in them in the first place. In the three years that I worked for the company this went from being the default position to being a completely foreign concept. By the time I left, the utter contempt that the moderators were held in (and bear in mind that 80% of my job was moderation) actually shocked me.

When they took the conscious decision to stop trusting their moderators and started implementing procedures to deal with this, they also started assuming that this now made them secure. What’s more, they then took this completely naive and false view and used it to implement even more procedures in a similar vein. To cap it all, they then started selling themselves as experts in this field.

“Nothing in the world is more dangerous than sincere ignorance and conscientious stupidity.” — Martin Luther King, Jr.

As a former social-psychologist I think that Groupthink is one of the things that still fascinates me most about all forms of the security industry (and I include warfare in this too). It’s almost like a purpose built Petri-dish for watching terrible concepts grow to such a size that nobody can see they are based on nonsensical premises which anybody who dares to question is immediately mocked.

“Доверяй, но проверяй.” — Ronald Reagan.

Part of the problem I have seen over many years is an obsession with not trusting your own people. I think there is a British/American divide here to a degree since I remember the big kerfuffle when the Americans discovered that the nuclear missiles on the British submarines were only secured by cheap bicycle locks. The experienced security people I know understand that the initial vetting and ongoing good treatment of staff is much more important than any amount of internal security. If Edward Snowden had really wanted to leak those documents then it’s unlikely that increased security measures would have stopped him. Security may be effective for stopping short term whimsical abuses but is no substitute at all for trust and respect in stopping long-term ones. The more oppressive and divided you make a work environment and the more obstacles you put in the way; the more you lose the respect of the workforce. In fact the more inviting you make yourself to exactly the abuses you are attempting to stop.

I saw another moderation company advertise that every keypress their moderators made was monitored in real-time by supervisors – I suppose this may be something approaching a solution but who would want to work for a company that did that? Do they expect to employ happy competent and experienced professionals?

“To forget one’s purpose is the commonest form of stupidity.” — Friedrich Nietzsche

Back to my original topic – My former employers; who out of politeness I will leave nameless. I will take one example of which they seem to be particularly proud: their secure login system. When I started at the company, they had a system that would allow the moderators to login to the various clients they worked on with a single icon press. It was nice and convenient and people sometimes used it. It wasn’t compulsory, it wasn’t made for high-security and when it was broken for some reason; the moderators would just ask somebody for the direct login passwords and they would use them instead.

Then they got a new technical/security manager, with no technical background at all and no experience at all in either security or moderation. He had a lot of bright ideas and talked the talk well enough though; it would seem.

They replaced the former login system because the old one was closing down. This wasn’t a bad idea and I still was with them so far. It was a nice convenience to have! But then they made the mistake of assuming this was a secure answer to all their prayers; and this is where their problems started.

For a start, the system they used (Onelogin) was only ever as secure as the people using and managing it, and the person managing it didn’t have a clue what he was doing. Although it is possible to setup secure conduits and exchange passwords securely this was never considered to the point where it took about 10 seconds to get a password out of the system. One of my supervisors asked me to write them a guide to “hacking” Onelogin so they could get the passwords out to give to their team in fact. A copy is available here if you are interested in just how simple it is.

“Stupidity is a talent for misconception.” — Edgar Allan Poe

But that’s just the start of it. As in all good Groupthink scenarios, the problem gets worse and builds on the original misconception. Access to client sites was done via a company proxy, and the Onelogin system was secured so it could only be logged into via this proxy and via the “secure browser” which all the staff now had to use. This could have been well and good, except that the proxy-setup was apparently the most unreliable thing in the entire universe. It wasn’t the proxy itself which had a moderately good uptime, it was the fact that the authentication system was next to useless. When it crashed (which it did, often), the technical manager had to be woken up to restart it. Obviously he didn’t trust anybody else to have access to it. This meant that often enough the proxy was was inaccessible for hours and hours at a time. The company was a 24/7 moderation company with some high profile clients, and still had some project managers who actually cared about them; so what would then happen is that they told the moderators to use another browser without the proxy enabled and they were given the passwords needed to moderate. They were then told not to tell any technical-people because that obviously wasn’t allowed.

So now you have a company who has put a lot of time and effort into creating a system that they are convinced is secure, and a staff-base, being unable to actually do the jobs for which they are paid and being relied upon to do; finding ways to circumvent all of this simply so that they can work. They aren’t doing this maliciously; they are doing this because the security in place is actually hampering them doing their jobs.

“Only two things are infinite, the universe and human stupidity, and I’m not sure about the former.” — Albert Einstein

One of the best things about all of this is that, now the company has done all of this; being social-media gurus as they are, they are telling the world all about it. They have not only done collaborative videos with Onelogin; they have also now set up a company to tell other people how to spread the misconceptions even further.

So folks – If you ever want to set up a 24/7 moderation company, with some very high-profile clients with sites which are constantly under attack by bad-folks who thankfully haven’t yet discovered the three completely separate and incredibly easy-to-attack single points of failure which would bring the entire company down with no means at all of switching to any alternatives.  If you ever want to completely alienate your work force so they have no respect or loyalty to you at all any more and if you ever want anybody with any technical background at all to look at you as though you are a naked-emperor on acid; then I know just the advisors for you!

Posted in Amusement, Groupthink, Rants | Leave a comment