URLTO GUIDE
Prepare publishing text with deduplication and counts
Text copied from several documents can contain repeated lines and inconsistent whitespace. Keep the source, clean the list and then compare its length with the destination's input rules.
URLTO editorial · Updated
Text counts · series01 / 10
1. Choose the deduplication options
Deduplication preserves the order of the first occurrence. Trimming makes ' apple ' and 'apple' match; without trimming they differ. Matching is case-sensitive, so 'Apple' remains distinct. Decide whether blank lines are meaningful before removing them.
Compare it yourself · Illustrative example
apple apple Apple banana → apple Apple banana
2. Distinguish characters from bytes
'가 A' contains one Korean character, one space and one Latin character. Character and UTF-8 byte counts serve different limits. Words are whitespace-separated groups, not language-aware linguistic token counts.
| Input | Characters | Without whitespace | UTF-8 bytes |
|---|---|---|---|
| 가 A | 3 | 2 | 5 |
| hello | 5 | 5 | 5 |
3. Review emoji and line breaks
Characters are counted as Unicode code points. Combined emoji can look like one symbol while containing several code points. A publishing service may use a different method, so validate near-limit text in its actual editor. Line breaks also affect character and byte totals.
4. Review the cleaned result before publishing
These tools process input text in the browser. Line matching does not merge sentences with similar meanings. Review order and omissions, then pass the result to the counter. Comparing it with the source helps avoid removing repetition that was intentional.