3

We have a site deployed in Norway, and people have uploaded PDF-assets with Norwegian non-ascii characters in the filenames.

Normally this works fine, the URLs are encoded like this and it works:

http://[oursite].no/assets/files/downloads/%C3%85rsberetning-2014_kladd.pdf

("Årsberetning" - annual report)

But one file didn't want to download, it only gave me 404. However, checking on the server the file was there, and the filename appeared correct. Also, re-scanning asset folders didn't remove it.

BUT - I was not able to rename the file from within craft, nor download it.

After much hair-pulling I established that the file on disk had the Å encoded as '\x41\xcc\x8a' whereas craft looked for a "pre-composed" Å ('\xc3\x85').

NOW the question - where did this break? :)

Does Craft unicode-normalise filenames? Does MySQL normalise the unicode it stores?

The file has moved around various developer machines and staging servers before making it into production, it could be some other part of the tool-chain that broke it.

  • I'm having problems with this all the time with norwegian characters. For me, the problem normally happends when moving files between mac osx and various linux servers. As long as the files is uploaded to craft to a server, and stay on that server, it works fine. – André Elvan Apr 16 '15 at 18:55
  • André - yeah, that is my experience as well. On our linux staging server the file was uploaded, there it worked fine (but the file was stored twice, with both encodings...) - then it went through git onto a osx machine, and from there it was pushed live with rsync. It's actually a miracle that anything ever works at all :) – gromgull Apr 17 '15 at 8:38
3

I think this is going to be one of those impossible to answer questions because of so many unknown variables.

MySQL could be screwing it up because it depends on 1) MySQL server configuration and 2) The charset and collation of the database, the tables, the connection strings, etc.

The file system itself could be screwing it up because they all tend to have different treatments for character encodings (HFS, NTFS, FAT, EXT, etc.).

Throw in the fact that you said this has been on various developer machines and touched by an unknown number of other programs, it makes it very difficult to diagnose.

| improve this answer | |
  • 1
    Given this, and I've requested this several times both for the EE and Craft versions of Craft, can we please, please, please have a setting to convert filenames to ascii, like slugs? :) – André Elvan Apr 16 '15 at 18:57
  • @AndréElvan Already on our internal list. :) – Brad Bell Apr 16 '15 at 19:02
  • Brad: I totally understand :) I just wanted to collect experiences from other people! – gromgull Apr 17 '15 at 8:36
  • I also take this to mean that "Craft does not intentionally apply unicode normalisation to filenames (or to anything)" - so it happened "somewhere else". – gromgull Apr 17 '15 at 8:38
  • 1
    @gromgull Correct... Craft isn't doing any filename normalization currently. – Brad Bell Apr 17 '15 at 15:57
4

This has now been solved by the addition of the convertFilenamesToAscii config setting introduced in Craft 2.4.

If you are working on a multilingual site, you might also find the limitAutoSlugsToAscii config setting useful.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.