Deep Fake Content: A New Reality Awaiting State Regulation

Justice might not come for women and girls from the Balkans, who have been slut-shamed and stalked through Telegram groups, mainly with so-called ‘deep fake’ photos of them being shared at unprecedented speed.

Early this year, a Telegram group called “AlbKings” that shared explicit photos and videos of women and girls without their consent was identified. The group had reached 120,000 members, mostly from Albania and Kosovo, but also from North Macedonia and other countries throughout Europe.

However, unlike previous groups in which sexually explicit content was shared, members in “AlbKings” also made use of artificial intelligence. Specifically, edited photos that have replaced a woman or girl in the original image with another by using an algorithm, known as ‘deep fake,’ were shared in this group. The ‘deep fake’ content posted to “AlbKings” was exclusively sexual or pornographic.

As technology advances at a rapid pace, the emerging trend presents serious threats, unmatched by the pace institutions respond. 

None of the eight Balkan countries define cyberbullying as a separate criminal offense, meaning that slutshaming must be punished under other criminal violations, such as gender-based hate speech, stalking, or as the unauthorised sharing of someone else’s photos. 

Hate speech based on gender or sexual orientation is a criminal offense in Montenegro, North Macedonia and Kosovo. However, Albania only classifies hate speech according to sexual orientation. In Montenegro, the criminal code classifies hate speech based on gender or sexual orientation only as “aggravating” factors.

In North Macedonia, the public prosecutor’s office is only now preparing amendments to the Criminal Code, which will include the misuse of persons’ recording, photographs, audio recordings or documents for the purpose of creating sexually explicit content.

“The Public Prosecutor’s Office will submit such a legal draft to the Ministry of Justice,” the Public Prosecutor’s Office told BIRN, noting that the existing legislation is not sufficient to adequately punish all crimes of this kind.

The majority of respondents from BIRN’s questionnaire rated abuse of personal photos as their dominant fear. 

“Undressing” photographs

The deepfake content in "AlbKings" was exclusively pornographic Photo: Screenshot from Telegram

The deepfake content in “AlbKings” was exclusively pornographic Photo: Screenshot from Telegram

Certain members of the “AlbKings” Telegram group profited financially by offering to use apps to make explicit content by “undressing” photos. The fake pornographic photos and videos were shared widely by the members of the group.

The group was closed and opened several times in the first half of the year. “AlbKings” was completely closed at the end of May 2024 when its seven administrators were arrested in Kosovo under charges of identity theft.

Most of the perpetrators from the “AlbKings” group evaded prosecution, while the administrators are being prosecuted for misuse of personal data. 

Before the police made the arrests, a new Telegram group with almost 50,000 members emerged. BIRN monitored the activities of this smaller Telegram group for weeks.

In the new group, members asked for numbers and photographs of girls and women from the Macedonian cities of Tetovo, Kichevo, Kumanovo and Skopje. 

The members did not target specific victims, but were asking for information about girls in their local areas. However, no photos of girls and women from North Macedonia were shared in “AlbKings” during the period that BIRN monitored the new, smaller group. 

“North Macedonia does not have specific legislation that regulates sexual violence through images. Rather, it is only partially and incidentally covered in other legal norms, such as misuse of personal data or unauthorized recording,” North Macedonia-based researcher Martina Draganovska Martinova told BIRN.

An example of this is the case against six minors from Kavadarci, which was initiated in February of this year. The perpetrators downloaded photos from the social media profiles of their classmates, with which they made photomontages that they shared with the goal to insult, ridicule and damage the reputation of the victims.

The perpetrators are being prosecuted under the criminal offense Misuse of personal data, which is punishable by a fine or imprisonment for up to one year.

Concerned and helpless

Most of the respondents fear that someone can misuse their personal photos Photo: BIRN

Most of the respondents fear that someone can misuse their personal photos Photo: BIRN

Although the Ministry of Interior told BIRN that they have yet to register a case of ‘deep fake’ pornography, the documented cases in other Balkan countries are cause for worry for the citizens across the region.

To investigate how present this fear is and how it has affected people’s behavior online, BIRN in North Macedonia published an anonymous questionnaire.

Most of the respondents fear that someone can misuse the personal photos they post on the Internet and have changed their behavior online accordingly.

Many respondents report that they post photos less often, and some have restricted access to their profiles or deleted their profiles on certain social media.

“I decided to make my profiles private, particularly Instagram since I share more pictures about my private life there. Especially because I have a daughter, I didn’t want just anyone to have access to the pictures I post of her, as well as of my extended family or friends,” Marta Stevkovska from Skopje told BIRN.

She explains that she began to fear for her safety on the Internet after the appearance of the Telegram group “Public Room” in which private numbers and profiles on social networks were shared in addition to explicit content.

When several men would follow Stevkovska on her profiles at the same time, her first thought was that someone might have misused her personal data or photos. Because of that, she closed most of her profiles.

In the case of “Public Room”, which in 2021 had over 6,500 members, indictments were brought against only a few people, while the group’s creator and administrator received four-year prison sentences for producing and distributing child pornography.

As most of the victims did not receive justice in North Macedonia, lawyer Natasha Boshkova told BIRN that she is now representing two victims from the “Public Room” group at the European Court of Human Rights.

“The ‘Public Room’ case is often celebrated as a victory, however, that is not really the case because all the data and pictures of the adult victims were not properly taken into account,” Sara Milenkovska, researcher and co-founder of the organization “Stella Network,” explained.

Slow procedures and mistrusting institutions

"This violence can take place in multiple jurisdictions," says Martina Draganovska Martinova | Photo: BIRN

“This violence can take place in multiple jurisdictions,” says Martina Draganovska Martinova | Photo: BIRN

This type of violence online is usually committed against women, however, some men anonymously responded to BIRN’s questionnaire about having been hacked or scammed. 

A man shared that his Facebook profile was hacked, leading to an unknown perpetrator sharing erotic content of women from his profile. He reported the case to the police and the Agency for Personal Data Protection, after which his Facebook account was closed. The incident led him to stop using Facebook.

Another male respondent said he was catfished by someone he met online, who had later blackmailed him to pay $500 (around 463 euros), or else photos they had shared on WhatsApp would be sent to the respondent’s friends. 

“I didn’t pay the money and disabled my accounts for 3-4 months. Their operations are abroad, so there was no logic in reporting the case to the Ministry of the Interior [in North Macedonia],” he wrote.

Civil society organizations and institutions point out an additional problem in situations like these is that large social networks and applications, such as Telegram, often do not always cooperate with the authorities.

“This violence can take place in several jurisdictions, and, thus, this complicates the task of determining who is responsible for prosecuting and sanctioning the perpetrators,” researcher Draganovska Martinova explained.

Another problem civil society organizations raise is that the employees in the institutions are often insufficiently sensitive to this issue and there is a lack of quick reaction on their part.

“The general impression is that the responsible institutions work slowly, are not sensitive to the gender perspective of these crimes, and, in that sense, do not provide adequate help and support to the victim, nor do they provide procedural protection during the process,” lawyer Boshkova told BIRN.

That is, the institutions do not always recognize these acts as a type of sexual violence and the victims, who are mostly women, are not protected. Boshkova has represented victims of this type of violence in court and says that for one of her cases, the prosecutor’s office decided that there was no evidence to initiate court proceedings in order to punish the perpetrator after three years of investigation.

Sometimes cases are taken over by other institutions, such as North Macedonia’s Agency for the Protection of Personal Data.

Complicated procedures, which often involve multiple institutions, can discourage victims and create space for perpetrators to re-traumatize the victim.

“While the victims wait for a resolution, they live in fear of what their abusers’ next step will be. They know that, unfortunately, all things remain on the Internet and that the violence does not end, but continues in the digital world,” concludes researcher Milenkovska.

After public pressure because of the “Public Room” case, the new criminal offense of “stalking” came into force last February, through which the perpetrators of digital sexual violence can be prosecuted.

Since then, North Macedonia’s Ministry of the Interior says that 17 crimes of “stalking” committed over the Internet have been registered, while 13 perpetrators were criminally charged.

Please enable JavaScript to view the comments powered by Disqus.