Normalizing the Chaos: Resolving Indonesian Twitter Slang in Customer Complaints

TL;DR

Social media text is a nightmare for standard NLP. This paper introduces a robust normalization pipeline specifically for Indonesian Twitter complaints, combining slang dictionaries with a Context Dictionary to resolve ambiguities. The result? A jump from 75% to nearly 90% accuracy in transforming "unstructured" tweets into machine-readable text.

Background: The Wild West of Indonesian Social Media

Indonesian internet culture is famous for its creative use of language, often referred to as Bahasa Alay. Users frequently:

• Replace vowels with numbers (e.g., "S4ya" for Saya).
• Drop vowels to shorten words (e.g., "Tdk" for Tidak).
• Use expressive repetition (e.g., "Maaaaaf" for Maaf).
• Use local slang (ciyus, boljug).

For businesses trying to automate complaint monitoring, this "free style text" acts as a wall that stops traditional sentiment analysis and data mining tools in their tracks.

The Problem & Motivation

While prior work (Sproat et al., Han & Baldwin) focused on general microblog normalization, they often lacked the linguistic nuance required for Indonesian affixes (Prefixes, Infixes, Suffixes). The authors recognized that for a machine to understand a complaint like "tgl di sini mkn hati," it needs to know if "tgl" means tanggal (date) or tinggal (live/stay).

The Insight: High accuracy requires more than a lookup table; it requires Context.

Methodology: The Three-Stage Pipeline

The authors propose a structured workflow to clean and normalize text:

1. Cleaning & Pre-processing

Filtering out '@' mentions, '#' hashtags, and links. It also handles "emotional" tokens like "hahaha" or "wkwk".

2. OOV Detection & Number Substitution

The system identifies Out-of-Vocabulary (OOV) words by comparing them against a Basic Word Dictionary (28,526 words) and a Slang Dictionary (378 words). A critical sub-step is Number Substitution, where "0" becomes "O", "3" becomes "E", and "2" is treated as a word repetition marker (e.g., "buku2" $\to$ "buku-buku").

3. Word Replacement & Context Resolution

This is the core innovation. If a word is ambiguous, the Context Dictionary—built from 2,323 word relationship pairs extracted from formal news complaints—is used to predict the most likely standard word based on neighboring tokens.

Figure 1: The proposed normalization architecture.

Experiments & Results

The researchers tested their algorithm on 910 tweets with the '#komplain' hashtag.

• Speed Optimization: Through code indexing and restructuring, the execution time for the dataset dropped from 3m 47s to a lightning-fast 1.84s.
• Accuracy Gains:
  • v1.0 (Baseline): 75.78%
  • v2.4 (Enriched Slang Dict): 82.17%
  • v3.0 (Full Pipeline + Punctuation Handling): 89.83%

Table: Evolution of performance across different versions of the algorithm.

Critical Insight & Analysis

The most impressive aspect of this work is the Context Dictionary. By utilizing a "clean" corpus (detik.com) to understand how words relate in a complaint context, the algorithm overcomes the "short-text" limitation of Twitter.

Limitations:

1. Code-Switching: The algorithm fails when Indonesian users mix in English words (e.g., "Service-nya bad bgt").
2. Language Evolution: Slang changes rapidly; the dictionary-based approach requires constant human updates to stay relevant.

Summary

This research provides a highly practical, computationally efficient framework for Indonesian text normalization. By bridging the gap between "Internet Slang" and "Formal Indonesian," it paves the way for more accurate automated customer service and social media sentiment tracking in the Indonesian market.