A TERF, or Trans-Exclusionary Radical Feminist, is the moniker given to a small subset of feminism by people who would really rather not be associated with them. TERFS have several ideological differences that alienate them from the rest of modern feminism, but the most glaring is their perpetuation of transphobia and transmisogyny.
The term was coined by Viv Smythe in 2008 on online Radical Feminist site. According to Smythe, the term was “meant to be a deliberately technically neutral description of an activist grouping. We wanted a way to distinguish TERFs from other RadFems with whom we engaged who were trans*-positive/neutral, because we had several years of history of engaging productively/substantively with non-TERF RadFems.” Despite the intentional neutral description, TERFs consider TERF to be a pejorative the same way racists don't like being called racists.
In short, a TERF is someone who considers themselves to be a Radical Feminist but does not consider transwomen, transmen, or the trans-community valid. Their general argument is that if you consider yourself to be a transwoman, then you are a self loathing gay man trying to muscle in on Women's Issues. If you are a transman, you are a self loathing woman who has been brainwashed by society to believe men are superior, and so you are either trying to join their ranks or are trying to distance yourself from femininity in a desperate and misguided move to protect yourself from oppression.
Much of TERF rhetoric in regard to the trans community echoes that of the Alt Right and other ultra-conservative neonazi scum, including the idea that being trans is a choice, that those who consider themselves trans are at best misguided, at medium mentally incapable, or at worst rapists-in-waiting. The hallmark of a TERF is the strict adherence that sex = gender, that there are only two sexes, and that gender/sex is determined solely by the fiddly bits you're born with-- regardless of any complications
What makes TERFs particularly vile is that, by operating under the banner of feminism, they have lobbied for practices such as reparative therapy and against transhealthcare or inclusion. It has become a common practice now for otherwise antifeminist conservative politicians to pass transphobic (either active or passive) legislation and getting an academic TERF to sign it off as feminist-approved. A notable early example of this would be Janice Raymond.
Janice Raymond, nun-turned-academic-lesbian-radical-feminist, who is otherwise known for her work against sex trafficking, in particular is credited with postponing the forward motion of trans-inclusive legislation. Specifically, she's the one the government looked to when researching trans medical care and whether or not it should be something an insurance provider could cover. In 1981 Raymond was commissioned and published by the National Center for Health Care Technology (NCHCT) to report on the ethical and financial effects of trans surgery and other medical care, which she deemed "[ethically] controversial, experimental, and expensive." In her report, she cites a review of her own book, The Transsexual Empire, a book whose choice quotes include: "I contend that the problem of transsexualism would best be served by morally mandating it out of existence" and "All transsexuals rape women's bodies by reducing the real female form to an artifact, appropriating this body for themselves .... Transsexuals merely cut off the most obvious means of invading women, so that they seem non-invasive."
(Those quotes weren't, as far as I know, actually in the NCHCT or OHTA reports. That was so you'd see where she was coming from.)
That report goes on to be used in another report by the Office of Health Technology Assessment (OHTA), and both reports were used by the Health and Human Services branch of the U.S. Government to evaluate whether or not to include trans-related medical care in assorted medical programs. In 1989, the HHS began revoking and banning trans-related medical care and instead taking up the rhetoric that transsexual/transgender individuals did not suffer from any medical issue, but mental ones, a move that wasn't undone until 2013. While Raymond claimed that she wasn't responsible for the legislation against trans medical care, the HHS Appeals Board that undid the legislation cited her NCHCT report as the reason for the initial 1989 ruling.
This sort of thing isn't uncommon. The most recent debacle would be the controversy regarding which bathrooms trans students are supposed to use in schools and how some vocal so-called feminists were at the forefront of calling transgirls potential rapists and perverts for wanting to use the girl's bathroom. Because I'm sure an outed transgirl using a boy's bathroom and locker room wouldn't lead to any kind of violence or bullying.
It's not like trans people are regularly targeted for harassment or anything.
https://en.wikipedia.org/wiki/Feminist_views_on_transgender_topics
http://transadvocate.com/fact-checking-janice-raymond-the-nchct-report_n_14554.htm
http://www.cristanwilliams.com/b/2016/09/01/janice-raymond-terfs-insurance-policy/
https://www.huffingtonpost.com/kelsie-brynn-jones/transexclusionary-radical-terf_b_5632332.html
https://rationalwiki.org/wiki/Trans-exclusionary_radical_feminism
Yes I know these aren't the most academic of sources, but I could hardly find any peer-reviewed articles of merit online free of a paywall. Besides, this is just a brief overview.