Fans of the show took to Twitter to complain about the service misrepresenting, censoring and simplifying dialogue from a variety of shows.
Mr Brown, the Queer Eye cast member who focuses on culture, said reading fan comments had broken his heart.
Tweets by Rogan Shannon, a deaf Netflix fan, in which he demanded that the service explain why it was not captioning word for word, have been shared thousands of times in recent days.
His tweets claim the subtitles censor profanity and edit dialogue for brevity.
Others accused the service of failing to caption foreign language inserts and correcting distinct dialects into Standard English.
Netflix has not yet responded to a BBC request for comment but after the outpouring of social media complaints said on Twitter that it was looking to fix some of the issues raised.
Subtitles are created in different ways by different broadcasters, with many employing outside subtitling firms. They can be written manually and time-coded to audio, or are generated using dictation software or audio recognition.
Gemma Rayner-Jones, 31, from Canterbury in England, uses subtitles to help her to concentrate when watching shows online because of a cognitive impairment.
Because she is able to hear and notice the differences, she has been tracking and complaining about inaccuracies in Netflix’s subtitles for about two years.
She estimates that she has submitted about 150 complaints in that time, and says she has not had a response.
“Everyone should be getting the same experience,” she told the BBC.
“It seems a shame to have a system to report faults there to placate people, but they don’t seem to be doing anything about it.”
She wants Netflix to be more transparent about how it handles complaints, so that users can check in whether action has been taken.
Student Chrissy Marshall, 18, studies film at the University of California and runs a YouTube account trying to raise awareness about deaf culture, accessibility and sign language.
She was one of many who took to Twitter to complain about inaccuracies in Queer Eye’s subtitling.
For her, online streaming still remains one of the best entertainment options available.
“I don’t watch cable or normal TV because captioning is always messed up or lagging. As for movie-going in theatres, the experience normally sucks,” she told the BBC.
“Netflix is what I use as a primary source for streaming because typically it is the most accessible, but even the most accessible has its issues.
“Captioning as a job is not to ‘clean up’ language, it’s to provide accessibility, full accessibility.
“We don’t care if it’s a bad word, vulgar, or maybe inappropriate, if hearing people get to know what is being said, we deserve to know as well.”
This is not an issue isolated to Netflix itself. While regulations are in place for closed captioning (user-activated) subtitles on typical television services, many on-demand services still lag behind.
One YouTube vlogger, Rikki Poynter, has dedicated years to working on accessibility on the platform, lobbying it to improve its automatic subtitle service using the hashtag #NoMoreCRAPtions.
In the US, the Federal Communications Commission has strict regulations which specify that captions “must match the spoken words in the dialogue and convey background noises and other sounds to the fullest extent possible” – but it only requires the regulations on shows on television, which means that Netflix-exclusive original series may not qualify.
The National Association of the Deaf sued and made a four-year agreement with Netflix in 2012, where it committed to ensuring all its programmes were subtitled.
Although the four-year decree has now run out, on Thursday the group told the BBC it was “disappointed that Netflix appears not to be providing captioning at the level that was promised” and said it hoped it would ensure it was using verbatim and accurate captions.
In the UK, Action on Hearing Loss has spent three years on a Subtitle It! campaign aiming to get the UK government to extend regulation to captioning of video-on-demand content.
Dr Roger Wicks, the group’s director of policy and campaigns, told the BBC that any attempt by providers to summarise or edit language on subtitles was a “very bad approach” which could lead to people who were deaf or hard-of-hearing feeling “alienated or patronised”.
“Subtitles are a replacement for speech, they’re meant to be verbatim so people have full access,” he told the BBC.
“Any attempt to summarise is offering a second-class service. I think this is well-intentioned, but it’s getting it wrong.”
He told the BBC his group intended to contact Netflix over the issue.
Mr Shannon, whose widely-shared tweets helped spark the debate, wants the company to change and check the way it subtitles its shows.
“I’d like to see more oversight on captioning agencies, more strict procedures for checking the captions,” he told the BBC.
“I’d also like to see those who are doing the hiring, such as Netflix, to check that all the files that they get are accurate, and not just assume they did everything right.
“I’m aware that it’s time consuming, but this will continue to be a problem if there are no checks and balances. Accessibility really matters.”